Thursday, February 2, 2012

Google's Privacy Policy

I've been building professional web applications for 12 years and everywhere I've worked, we built the most complete profile of our users that we possibly could. There is really no other way to operate. When there is an issue, the first question is, "How many users are affected?" You need to know as much as possible about the user in order to answer this question. Usually, that starts with their browser, operating system, and IP address.

The second question when an error occurs is, "How did this happen?" For that, you need a history of everything every user has done. Web applications are not the only ones who collect or use this information. Your operating system, browser, the domain name system, your ISP, and every hub and router that your communications pass through has to have some level of logging, auditing, or tracking in order to function correctly (this logging may normally be turned off, but it's there, or can be installed). Unless you break into someone's computer, the web has never been anonymous. Something like Tor can create some anonymity. If you are really concerned, use Tor.

Last night, someone on On Point suggested using Yahoo! mail instead of Google mail, but Yahoo!, Twitter, Amazon, Microsoft and others are trying to collect every bit as much information about you as Google is. The only reason that Google might collect more is that Google is just doing a better job of capturing everything.

I think what really upsets people is that Google has been so successful at using this information to target ads for people. Without the targeted ads, the information collection is not visible to the end-user. All of a sudden, it seems like Google and Facebook are doing this terrible thing, when really, everyone is doing it and you only see the results when you use those products.

Having paid to advertise using Google Adwords, I can say without a doubt that when Google shows you an ad for Type II diabetes, they are not telling the advertisers that you have Type II diabetes. The advertiser does not know anything about who you are, just that someone, somewhere used the word "Diabetes" and that Google showed your ad, and whether or not someone clicked on it. Google has a program that sees the word "Diabetes" in your profile and that fires ads that also have the word "Diabetes" in them. They are not disclosing anything about you to advertisers.

I joined Audubon last year, and within 2 weeks, I had requests for donations in my mailbox from The Sierra Club, The Arbor Day foundation, and half a dozen other environmental organizations. There is something very ironic about receiving a pulped, dead tree (paper) in the mail from the Arbor Day foundation. I felt mildly violated, knowing that my gift to Audubon resulted in them giving my address to a bunch of other organizations, who I then had to call up and ask to be removed from their mailing lists. Where is the outrage against these organizations? Why the outrage against Google and Facebook, who have never given my street address or email address to anyone?

If you share your Google login with someone else, or use your computer where someone can see your screen, how is Google supposed to protect you from that? Tell other family members to get their own profile. If you don't want to see adds for divorce lawyers, don't discuss divorce with friends in email - use the phone or talk in person. People don't read porn in public. Why should they expect to surf web sites in public that they don't want other people to know about?

I cannot imagine any meaningful way to make the web truly anonymous that does not also give hackers and spammers a huge edge over legitimate web sites and e-commerce. The profiling and tracking that every major web presence uses today is necessary to prevent spam from dominating our inboxes and forums, and also to catch hackers when they break in and steal or destroy things. True anonymity on the web, were it possible, might make e-commerce impossible and email unusable due to unstoppable vandalism, theft, and spam.

Most of the time, targeted ads are a good thing. If you have a nun and a gigolo, doesn't it make sense to show an ads for habits to the nun and condoms to the gigolo? Do you really want the reverse of that happening? If I'm going to have to look at ads, I'd rather look at ones relevant to me than irrelevant ones.

The targeted ads that are upsetting people so much are what pay for Gmail, YouTube, Facebook, and most of the sites people frequent on the web. If you don't want Google ads, you can pay them $50/year for Google Apps. For free, you get targeted ads. Live with it.

Monday, January 9, 2012

Ubuntu 11.10 "Oneiric" with a Gnome 2 Feel

I love, and am maybe even a little addicted to the packages Ubuntu includes in their distributions. But like many, I am dissatisfied with the new Unity interface because:

  1. I can't open two copies of KeePassX at once
  2. I hate using text search to find applications
  3. It was too hard to create a custom application launcher
Hopefully all that will change, but until then, it's just not working for me.

I tried the LXDE desktop, but:

  1. I didn't like the default software (Leafpad text editor, Galculator, LX terminal)
  2. Desktop integration is lacking (no screen-shots, limited drag and drop from one application to another)
It's a very new distribution and may need a year or two to catch up.  Certainly LXDE is worth keeping an eye on, but it's not there yet.

What's currently working for me is a "retro" basic Gnome session based on OMG Ubuntu's article, Make Ubuntu 11.10 Look and Feel Like GNOME 2. Thanks to "DigalMan" for pointing it out to me. That article has a typo though, it's gnome-session-fallback, not gnome-fallback-session.

Also, when they say, "Add ‘ppa:jconti/gnome3‘ to your Software Sources" their instructions don't include how to add the PPA's key, so that future updates are not be applied properly (or at all). Fortunately, launchpad.net has instructions. The key lines I needed were:

sudo add-apt-repository http://ppa.launchpad.net/jconti/gnome3/ubuntu
sudo apt-get update

It's not perfect (the top-bar color's not quite right and the icons space themselves wider apart when hovered over), but I'm basically back with the Ubuntu system I have loved for years, with the latest versions of all software. If this works in 12.04, I'll probably stick with it. If not, and LXDE has not improved by that time, I'll guess I'll try Mint.

Wednesday, September 14, 2011

Automatic POJO Generation From a Database

Like JPA (or presumably Spring), Hibernate "reverse engineering" tools can generate POJOs (Plain Old Java Objects) from database tables and vice-versa. Generating database tables from Java code is probably best used as a one-time short-cut, suitable for rapid prototyping. Because everything in an application is dependent on the database (and not vice-versa), future changes must be made in the database first (and any existing data migrated there first as well), then propagated to all affected parts of the application.

I have found that anything which the "database reverse engineering" process does not generate for me breaks, usually sooner rather than later. Also that the hardest part of managing maintenance of a large system is the constant refactoring. To that end, I have developed 2 goals for the database reverse-engineering process:

  1. It should keep my application in synch with any database changes, automatically updating as much of my application as is practical.
  2. Where #1 is not possible, it should cause any affected areas of my application to generate a compile-time error.

Imagine a database table:

CREATE TABLE `user` (
  `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
  `last_modifier_id` bigint(20) unsigned DEFAULT NULL,
  `company_id` bigint(20) unsigned NOT NULL,
  `identifier` varchar(64) NOT NULL COMMENT 'PIN.  Unique within a company',
  `first_name_c` varchar(40) DEFAULT NULL,
etc.

Database generation tools normally generate the following sort of fields:

public class User implements java.io.Serializable {

private long id;
private User lastModifier;
private Company company;
private String identifier;
private String firstNameC;
etc.

public long getId() { return id; }
public void setId(long x) { id = x; }

public User getLastModifier() { return lastModifier; }
public void setLastModifier(User x) { lastModifier = x; }

public Company getCompany() { return company; }
public void setCompany(Company x) { company = x; }

public String getIdentifier() { return identifier; }
public void setIdentifier(String x) { identifier = x; }

public String getFirstNameC() { return firstNameC; }
public void setFirstNameC(String x) { firstNameC = x; }
etc.

I have modified the generator to also generate the following fields for each column in the table (prefixed with SQL_) and for each field in the Java object (prefixed with HQL_):

public static final String SQL_user = "user"; // table name
public static final String SQL_id = "id";
public static final String SQL_last_modifier_id = "last_modifier_id";
public static final String SQL_company_id = "company_id";
public static final String SQL_identifier = "identifier";
public static final String SQL_first_name_c = "first_name_c";
etc.

public static final String HQL_User = "User"; // class name
public static final String HQL_id = "id";
public static final String HQL_lastModifier = "lastModifier";
public static final String HQL_company = "company";
public static final String HQL_identifier = "identifier";
public static final String HQL_firstNameC = "firstNameC";
etc.

This allows me to write code like the following:

Crit<User> dupIdentCrit = Crit.create(User.class);
dupIdentCrit.add(Restrictions.eq(User.HQL_company, this));
dupIdentCrit.add(Restrictions.eq(User.HQL_identifier, identifier));
List<User> dupIdentifierUsers = dupIdentCrit.list();

Here's an example of building a SQL statement for use with JDBC:

StringBuilder sqlB = new StringBuilder("select * from " + User.SQL_user +
                                       " where " + User.SQL_company_id +
                                       " = ?");
if (!includeDeleted) {
    sqlB.append(" and " + User.SQL_is_deleted + " is false");
}
sqlB.append(" order by " + User.SQL_last_name_c + ", " +
            User.SQL_first_name_c + ", " +
            User.SQL_middle_name_c);

When the database changes and I rebuild my objects, I get compile-time errors, showing me every piece of code I need to fix.

I am working on modifying the generator to also generate the following for any varchar or char fields because their maximum length is defined in the database columns:

public static final int MAX_identifier = 64;
public static final int MAX_firstNameC = 40;
etc.

I am considering parsing comments on each column to look for something like: 'min: 6' and generate the following from it:

public static final int MIN_identifier = 6;

That same technique could be used with int columns to define minimum and maximum values which could be used in validation, in the GUI, and in the help of an appliction. I'm also experimenting with using the HQL_ tokens as the names of the input fields on the GUI screens.

I'm not sure how or even if this would work with stored procedures.

Special thanks to J. Michael Greata, whose interest and suggestions over the past several years have encouraged and shaped the development of this project. Also to Arash Tavakoli whose post in the LinkedIn Java Architects group inspired me to organize my thoughts into this posting.

Thursday, September 8, 2011

Checked vs. Unchecked Exceptions

Two-Sentence Review: Checked exceptions have to be declared in the method signature and dealt with by the calling code; Unchecked, RuntimeExceptions don't. The Java compiler enforces this rule with a compile-time error.

I've been enjoying, "Java: The Good Parts" by Jim Waldo and just finished chapter 3: "Exceptions." At the end of the chapter, Mr. Waldo takes a humorously firm stance against RuntimeExceptions. Indeed, RuntimeExceptions can be evilly misused. Yet I believe there is a time and a place for these exceptions. Knowing when to use them requires an understanding of who is at fault for a particular problem, and whether the problem is recoverable or not.

The following code sample bit me the other day:

// EVIL - NEVER DO THIS!
public static void write(String s) {
    try {
        out.write(s);
    } catch (IOException ioe) {
        throw new IllegalStateException();
    }
}

It is evil (as Mr. Waldo says) for several reasons:

  1. This method is not being responsible for handling its own problems. IOExceptions happen for good reasons other than coding errors - users can delete files, network connections can close, etc. Better than hiding this exception this method might retry the write() or recover some other way. If recovery were not practical, this method would do better to let the original exception bubble up to the caller. Another improvement might be to have this method report the error some other suitable way (e.g. to the user).
  2. Wrapping a checked exception with a RuntimeException means that this method signature is missing vital information that the caller really needs to know about. It hides critical details (any problem with the write() now causes an unexpected exception).
  3. Absentmindedly wrapping an exception with another exception serves only to complicate the stack trace - it further hides the cause of the problem. It's good to wrap an exception if you have critical information to add (e.g. the method handles two streams and you wrap an exception with the message of which stream it applies to). In the example above, nothing is added. In fact, this code doesn't wrap an exception, it throws it away and substitutes another which is even worse.
  4. The IllegalStateException is blank, giving the unfortunate caller no idea what went wrong.

The following shows proper usage of RuntimeExceptions:

/** Creates or returns an existing immutable YearMonth object.
@param year a valid year
@param month a one-based month between 1-12
@return the relevant YearMonth object
*/
public static YearMonth valueOf(int year, int month) {
    if ( (month < 1) || (month > 12) ) {
        throw new IllegalArgumentException("Month must be a positive" +
                                           " integer such that 0 < n < 13");
    }
...

This is good because:

  1. Valid input values are documented clearly for the caller in the JavaDoc.
  2. The inputs are checked at the beginning of the method, before any processing is done, to fail-fast and fail-loudly if passed garbage.
  3. It throws a RuntimeException to inform the caller that they have made a coding error.
  4. The exception provides information about what the caller did wrong.

You wouldn't want to use a checked exception because it would force the responsible caller's code to check their input values twice:

YearMonth ym;
// Responsibly check inputs before calling valueOf()
if (m > 12) {
    out.write("Month too big");
} else if (m < 1) {
    out.write("Month too small");
} else {
    // I already checked my inputs.  Why do I have to check for an exception
    // too?  Doing so doesn't benefit me in any way. It just makes the
    // interface unnecessarily difficult to use!
    try {
        ym = YearMonth.valueOf(y, m);
    } catch (Exception e) {
        out.println("month still invalid: " + e.getMessage());
    }
}

The critical nuance here is that RuntimeExceptions should be used to indicate a programming error on the part of the caller of the function that throws it. They are generally used in the first few lines of the function as a defensive check for invalid input values. Each RuntimeException should include a description of the problem (not be blank).

RuntimeExceptions can also be used to catch invalid state as follows:

enum Numb { ONE, TWO; }
private Numb num = null;

public void init(Numb n) {
    if (n == null) {
        throw new IllegalArgumentException("init cannot take a null Numb");
    }
    num = n;
}

public void showNum() {
    if (ONE == num) {
        out.println("1");
    } else if (TWO == num) {
        out.println("2");
    } else {
         throw new IllegalStateException("Unhandled value of Numb or called" +
                                         " showNum without initializing the" +
                                         " num");
    }
}

When some programmer adds THREE to Numb and doesn't account for the new possible value, it fails hard and fast, making the problem easy to find and fix.

RuntimeExceptions are a good way to make code fail hard and fast without forcing the caller to check their input values twice. Used properly they can make sneaky coding errors obvious. Used improperly, they can make and obvious errors sneaky.

Wednesday, August 31, 2011

Object Mutability

This post is part of a series on Comparing Objects.

I've read a lot lately about making objects immutable whenever possible. "Programming in Scala" lists Immutable Object Tradeoffs as follows:

Advantages of Immutable Objects

  1. Often easier to reason about because they do not have complex state.
  2. Can be passed freely (without making defensive copies) to things that might try to modify them
  3. Impossible for two threads accessing the same immutable object to corrupt it.
  4. They make safe Hashtable keys (if you put a mutable object in a Hashtable, then change it in a way that changes its hashcode, the Hashtable will no longer be able to use it as a key because it will look for that object in the wrong bucket and not find it).

Disadvantages

  1. Sometimes require a large object graph to be copied (to create a new, modified version of the object). This can cause performance and garbage collection bottlenecks.

For most purposes, an object representing a month can be made immutable - February 2003 will never become anything other than what it is. But a User record is not immutable. People get married or change their name for other reasons. Phone numbers, addresses, hair color, height, weight, and virtually every other aspect of a person can change. Yet the person is still the same person. This is what surrogate keys model in a database - that everything about a record can change, yet it can still be meaningfully the same record.

In order to use an object in a hash-backed Collection (in Java), its hashcode must NOT change. The simplest way to accomplish this is to make the hashcode of a mutable persistent object its surrogate key and to use that key as a primary comparison in the equals method as well (see my older post on Implementing equals(), hashcode(), and compareTo()).

To make an immutable object, you sometimes need a mutable constructor object, like StringBuilder and String. StringBuilder allows you to change your object as many times as you want, then get an immutable version by calling toString(). This is clean and safe, but has some small costs in time and memory (transforming the immutable StringBuilder into a new immutable String object, then throwing away the StringBuilder). An alternative that I have not seen much is to create an immutable interface, extend a mutable interface from it, and then extend your object from that.

Here's an example based on java.util.List. Pretend each interface or class is in its own file:

// All the immutable-friendly methods from java.util.List.
// Interfaces like these could easily be retrofitted into
// the existing Java collections framework
public interface ImmutableList {
    int size();
    boolean isEmpty();
    boolean contains(Object o);
    Iterator<E> iterator();
    Object[] toArray();
    <T> T[] toArray(T[] a);
    boolean containsAll(Collection<?> c);
    boolean equals(Object o);
    int hashCode();
    E get(int index);
    int indexOf(Object o);
    int lastIndexOf(Object o);
    ListIterator<E> listIterator();
    ListIterator<E> listIterator(int index);
    List<E> subList(int fromIndex, int toIndex);
}

// This interface adds the mutators
public interface java.util.List extends ImmutableList {
    boolean add(E e);
    boolean remove(Object o);
    boolean addAll(Collection<? extends E> c);
    boolean addAll(int index, Collection<? extends E> c);
    boolean removeAll(Collection<?> c);
    boolean retainAll(Collection<?> c);
    void clear();
    E set(int index, E element);
    void add(int index, E element);
    E remove(int index);
}

public class java.util.ArrayList implements List {
    // just as it is now...
}

public class MyClass {
    someMethod(ImmutableList<String> ils) {
        // can't change the list
    }

    public static void main(String[] args) {
        List<String> myStrings = new ArrayList<String>();
        myStrings.add("hello");
        myStrings.add("world");
        someMethod(myStrings);
        // Totally safe:
        System.out.println(myStrings.get(1));
    }
}

This doesn't solve the problem of passing a list to existing untrusted code that might try to change it. It also doesn't prevent the calling code from modifying myStrings from a separate thread while someMethod() is working on it. But it does provide a way (going forward) for a method like someMethod() to declare that it cannot modify the list. The programmer of someMethod() cannot compile her code if she tries to modify the list (well, short of using reflection).

Guaranteed immutability can be critical in writing concurrent code and for keys in hashtables. Not all objects can be made immutable, but many of those objects have immutable surrogate keys that, if used properly, work around the pitfalls of mutability. Limiting mutability and avoiding common mutable object pitfalls can lead to fewer bugs, easier readability, and improved maintainability.

Thursday, August 25, 2011

Typesafe List in Java 5+ part two...

Looks like John's comment on yesterday's post had us both up half the night working on the same thing. I'm posting my version here because it has a parameter that lets the caller determine the trade-off between speed and safety. When debugging, you can use Check.ALL (or leave it null). In performance-critical code, you can later set it to NONE. Check.FIRST is for situations where you just a sanity check - it's fast and better than nothing.

I've also included some test code and a routine that takes advantage of the String.valueOf() methods to coerce each element of the original list to a String, if necessary, much the way that many dynamically typed languages do. I wonder if it would be worthwhile to further optimize this method for long lists by dividing the original list into runs of items that do not need casting and copying those runs with stringList.addAll(list.subList(fromIdx, toIdx)) instead of copying one item at a time?

John's solution uses B.class.isAssignableFrom(a.getClass()) while mine uses a instanceof B. I think either works in this case because a List can only hold objects, not primitives. If we were dealing with primitives, John's solution would handle more cases than mine. Not sure if there are performance considerations, but I doubt they would be significant.

import java.util.ArrayList;
import java.util.List;

public class TypeSafe {

    public enum Check {
        NONE,
        FIRST,
        ALL;
    }

    @SuppressWarnings({"unchecked"})
    public static <T> List<T> typeSafeList(List list,
                                           Class<T> clazz,
                                           Check safety) {
        if (clazz == null) {
            throw new IllegalArgumentException("typeSafeList() requires a non-null class parameter");
        }
        
        // Should we perform any checks?
        if (safety != Check.NONE) {
            if ( (list != null) && (list.size() > 0) ) {
                for (Object item : list) {
                    if ( (item != null) &&
                         !(item instanceof String) ) {
                        throw new ClassCastException(
                                "List contained a(n) " +
                                item.getClass().getCanonicalName() +
                                " which is not a(n) " +
                                clazz.getCanonicalName());
                    }
                    // Should we stop on first success?
                    if (safety == Check.FIRST) {
                        break;
                    }
                    // Default (Check.ALL) checks every item in the list.
                }
            }
        } // end if perform any checks
        return (List<T>) list;
    } // end typeSafeList()

    public static List<String> coerceToStringList(List list) {
        if (list == null) {
            return null;
        }
        try {
            // Return the old list if it's already safe
            return typeSafeList(list, String.class, Check.ALL);
        } catch (ClassCastException cce) {
            // Old list is not safe.  Make new one.
            List<String> stringList = new ArrayList<String>();

            for (Object item : list) {
                if (item == null) {
                    stringList.add(null);
                } else if (item instanceof String) {
                    stringList.add((String) item);
                } else {
                    // If this throws a classCastException, so be it.
                    stringList.add(String.valueOf(item));
                }
            }
            return stringList;
        }
    } // end typeSafeList()

    @SuppressWarnings({"unchecked"})
    private static List makeTestList() {
        List l = new ArrayList();
        l.add("Hello");
        l.add(null);
        l.add(new Integer(3));
        l.add("world");
        return l;
    } // end makeTestList()

    public static void main(String[] args) {
        List unsafeList = makeTestList();
        List<String> stringList = coerceToStringList(unsafeList);

        System.out.println("Coerced strings:");
        for (String s : stringList) {
            System.out.println(s);
        }

        List<String> safeList = typeSafeList(unsafeList,
                                             String.class,
                                             Check.ALL);

        System.out.println("Safe-casted list:");
        for (String s : safeList) {
            System.out.println(s);
        }
    } // end main()
}

Wednesday, August 24, 2011

Checking an Unchecked Cast in Java 5 or Later

If query.list() returns an untyped List, then the following code will produce an ugly warning: "Unchecked Cast: 'java.util.List' to 'java.util.List<String>'
List<String> tags = null;
try {
    ...
    tags = (List<String>) query.list();
    ...
Unfortunately, you can only use @SuppressWarnings on a declaration. I don't want to add this annotation to my method declaration because my method does a lot more than this one cast and this is a useful warning to detect in the rest of the method. One solution is to add a new variable to make a declaration to use @SuppressWarnings on:
List<String> tags = null;
try {
    ...
    @SuppressWarnings("unchecked")
    List<String> tempTags = (List<String> query.list();
    tags = tempTags;
    ...
Now it produces an inspection warning in IDEA 10.5, "Local variable tempTags is redundant." I should probably just ignore this warning or turn it off altogether, but I wasn't quite comfortable with that either. In situations like these, I naturally turn to Joshua Bloch for guidance and his item #77 - Eliminate Unchecked Warnings has this advice in bold:
If you can't eliminate a warning, and you can prove that the code that provoked the warning is typesafe, then (and only then) suppress the warning with an @SuppressWarnings("unchecked") annotation.
I could probably prove that my query is only going to return Strings, but it occurred to me that there is a more general way to satisfy all the above requirements:
@SuppressWarnings({"unchecked"})
private List<String> getListOfStringsFromList(List list) {
    if ( (list != null) && (list.size() > 0) ) {
        for (Object item : list) {
            if ( (item != null) && !(item instanceof String) ) {
                throw new ClassCastException("List contained non-strings Elements: " + item.getClass().getCanonicalName());
            }
        }
    }
    return (List<String>) list;
}
The method does nothing but make a typesafe cast on a list, so I can @SuppressWarnings on the whole thing. It throws an undeclared runtime ClassCastException, but only if the caller makes a programming error and this exception would get thrown anyway wherever the list was eventually used if I didn't throw it here. In short, this method takes a programming error that cannot be caught by the compiler and does the next best thing: fail-fast, making the error easy to find. Using the above solution hides all that complexity and preserves the intent of the original line of code:
List<String> tags = null;
try {
    ...
    tags = getListOfStringsFromList(query.list());
    ...
I thought this was interesting enough to post. Well, at least interesting enough for people who are considering a Java logo tattoo. If you liked this, you probably want to check out John Yeary's Response and my Updated Generic Version (part 2 of this post).