First evaluation result

I have a (pretty terrible) result: with the way things stand right now, tracSnap is able to predict 16% of the people who comment on tickets related to Hadley model defects.

How I arrived at this result:

1. Jon gave me a list of tickets (defects) that he was able to associate with a specific revision of the repository.
2. For each such ticket, I looked at the files that were involved in its associated revision, and used tracSnap to get the "experts" for those files.
3. I compared this list of experts with the people who actually helped to fix the defect, and checked to see if they are the same people.

I have a hunch that I can improve this statistic by finding a better way to relate changes in the branches to the files in the trunk. Right now, if you make changes in your branch, you are an 'expert' on only the files in your branch, and not the corresponding file in the trunk. So if you aren't the one to merge your changes to the trunk, you never get associated with the trunk files, and thus will not get suggested as an expert.

As of yet, I've only looked at the prediction rate using the "experts", and not the ticket reporter's "suggested contacts". I suspect that both analyses will produce similar results, since the suggested contacts and experts are not independent (contact suggestions rely partially on the expertise calculations).

16%... lame.

-- Edit: November 24 --

The 16% was the recall percentage. Precision seems to be 63%. That's a happier number.