Palm Based Scoring at the
2003 Area
7 Championship
August
29-31, 2003
Rob
Boudrie L1571
The purpose of this paper is to document our experiences using Palm based scoring for the 2003 Area 7 Championship, to assist others attempting or considering similar projects. This information is presented “warts and all”, and includes all the glitches, problems and corrective actions necessary during that event.
Harvard Sportsmen’s Association (NE03), Tom Dings TY41365 and Rob Boudrie L1571 all received free copies of the ASS scoring program. Beyond that, they have no financial interest in the product or its sales.
· Things went better than expected, and I consider this experiment to be a resounding success.
· This system can produce better results, quicker and with less work than paper scoring.
o Disclaimer: We did not verify the ability of stats officer to enter the correct scores into the Palms – but for that matter, there has never been a controlled double-blind study to analyze the error rate in transcribing called scores to paper score sheets.
· There were a few “first time” issues which had to be resolved by the “back room” stats team, but nothing which interfered with the collection of match data. Lessons learned in this match, combined with improvements to the Palm scoring software, will make things go even more smoothly next time.
· At this point, I recommend that matches maintain a paper backup of scores “just in case”, and to provide a “signed score” in the event of competitor disputes to posted scores.
· I recommend that stats be run by a competent computer person familiar with the workings of EzWinScore, Palm, and the MS/Access ODBC connection. I expect this recommendation to be relaxed in the future as the PalmóEzWinScore interface further evolves.
· I will use Palm/ASS scoring again for a major when I have the opportunity. Harvard Sportsmens Club plans on scoring their local 2004 match season using Palms, ASS and EzWinScore.
The stats crew consisted of Rob Boudrie and Tom Dings
We had 10 Palm IIIxes – One for each of 8 stages, one “Master” and one “spare”. We also had two Windows 2000 Laptops, but did all score entry on one and used the other to drive a printer.
ASS – Automated Scoring Systems, www.autoscoringsystems.com - the company run by Peter Cunningham which offers PSS, USPSS and the transfer utility.
USPSS – Palm Scoring adapted for USPSA rules. We used this at the match.
PSS – Palm Scoring System, international version. We did not use this.
Transfer program – A MS/Access based utility which transfers scores from the Palm to EzWinScore via the Palm docking port.
The Harvard Sportsmen’s Association (Area 7 host club) tested the Palm scoring system on two stages of a local match a couple weeks prior to the Area 7 Championship. Based on that successful test, we decided to go ahead and use the Palms on all stages at Area 7. The plan was to attempt to use them on the “staff day” (Friday), and use them for the entire match if Friday use went well.
We finished match setup Friday morning, had lunch, and then met for an RO meeting in the clubhouse prior to the match. I handed out one Palm IIIxe to each CRO, and gave 15 minutes of instruction on how to score shooters. This instruction consisted of taking the RO’s through entry of a score for a “dummy shooter” who was in the EzWinScore database, but removed prior to publication of results. ROs were instructed that Palm scoring was not to be allowed to hold up the match. If they had any problem, they were to first radio stats for help and immediately move to paper (without waiting for stats to arrive). All scores from the Palm were to be entered onto the shooter’s score sheet in “total form” (number of hits, penalties and time), to provide a paper backup and initialed shooter acceptance of the scores. These would be easily identifiable as Palm score sheets by the empty individual target boxes.
Match results were done in record time, with relatively few “posting period” corrections. The range officers performed admirably, and exceeded our expectation in “Palm Competence” with minimal training. Paper score entry was kept at the 1% (ONE PERCENT!) level; 99% was done on Palms.
1. Stage ID Problem – ASS added a feature to the PSS at my request to allow a Palm to be “locked” to an individual stage, in order to simplify the use of an individual Palm as an individual stage score entry system. The “locking” of a stage changed an internal stage code used to identify stages when beaming to the master in USPSS version 2.34. As a result, we were unable to beam scores to the master Palm on Friday. There was no loss of data however, as we were able to import scores from each individual Palm into EzWinScore without difficulty. I contacted Australia Friday evening, and a fix was delivered (version 2.36) prior Saturday’s start. This fix totally resolved the issue, and we were able to do repeated “sync to Master Palm” operations throughout Saturday and Sunday. In summary – we lost no data, and this problem has been fully resolved for future matches.
2. User error zeroed score on transfer – PSS contains a feature used to transmit an individual score (so that the RO can give a shooter his scores immediately). One of the stats runners invoked this transfer instead of the “beam all scores” function when picking up scores. At the time this was done, the Palm was in score entry mode for the next shooter, and a “zero” score beamed for that individual. These “zero scores” (there were only about 5) were easily identified in stats, and recovered from the paper score sheet. One was discovered during the posting period; all others were found and corrected by stats. ASS (version 2.37, which was released after the match) was enhanced so that an “empty” score cannot be beamed, and the messages will clearly indicate when you are beaming an individual score as opposed to all stage scores.
Post-match examination confirmed that the actual shooter scores were in the Palms (as the real scores over-wrote the zeros), but were not transferred to EzWinScore since we were using the transfer utility in “non-overwrite” mode (to allow manual entry of paper re-shoots on EzWinScore) to over-ride any Palm scores. It was our choice to do use this procedure so that a paper score would always take precedence over a Palm score (in case there was a paper based re-shoot which was to replace a Palm based original stage score).
In summary –
a. We had to obtain a few scores (less than 10) from the paper score sheets, but no data was lost.
b. This was, in part, “user error” at the match
c. The actual scores were present in the Palms, but not downloaded due to our selection of transfer modes (the original “zero” score was not over-written).
Changes made to USPSS after this match now prevent the transmission of zero scores which do not have a time entered.
3. EzWinScore DQ Problem – We DQ’d two shooters mid-way through the match by using the EzWinScore feature. Subsequent data imports from ASS resulted in some stages which were not marked as DQ, resulting in a non-zero match point total for these two disqualified shooters. I do not consider this a Palm scoring bug, since EzWinScore has the same problem if you import scores from an EzWinScore slave machine which has not DQ’ed all shooters DQ’ed in the master EzWinScore system. I resolved this issue by Un DQ’ing and re-DQ’ing these shooters in the EzWinScore system after all imports were done. This was caught by myself during the posting period. With advance knowledge, this issue is easy to avoid.
4. Scores not saved – We encountered a few scores (less than 10) which were on paper in “hit total only” form (indicating they had been entered into a Palm), but were not in the Palm database. I believe this was “user error”. All scores were recovered from paper.
5. Changing a stage in the Palm at the last minute – I had prepared all Palms in advance of the match. Once at the match, we discovered that one stage was one hit per target, not two as we had configured it in the Palm and EzWinScore. We changed the stage configuration for the Palm used at that stage and the EzWinScore stage configuration. We subsequently discovered that Palm score import requires the entire “match/stage configuration” to match between Palm and EzWinScore, so the scores would not import from the other Palms. This was easily remedied by correcting the stage configuration on the other Palms. Interestingly enough, the only reason this issue became relevant was out need to transfer scores directly from each Palm to EzWinScore (without use of a Master Palm) – see item #1 above.
6. Dummy or disabled stages in EzWinScore – We had 8 stages at this match, but wished to print 10 competitor labels per shooter. In order to do that, I “tricked” EzWinScore by creating two dummy stages each of which was named “Spare Stage”, bringing the stage total to 10 in EzWinScore, after which I printed the stage labels and disabled these stages in EzWinScore. EzWinScore does not permit total “removal” of a stage, but does allow you to mark it as not part of the match. The problem here is that the Palm system still looks at these stages when making sure that match stages match. This problem was easily remedied by adding the same two dummy stages to the Palms.
7. Fixed Time Stages – The transfer program does not handle fixed time stages at this time. This was not an issue at Area 7 as there were no such stages.
8. Wrong shooter number entered – When competitor 150 stepped to the line, competitor number 15 was entered into the Palm instead of 150 by the RO. This was caught by a competitor at an interim posting, and would have been found by stats when Shooter 150’s score came up “missing”. Recommendation: Be sure shooters and range officers verify the competitor name on the Palm before entering scores.
9. Last
name insufficient unique ID for competitor verification – The Palm software at the match
displayed only the last name of the competitor on the score entry screen. This meant that a member of a father/son
team could not tell for certain the score entry panel was their own. ASS has already responded by changing this
display to the first 8 characters of the surname followed by the first three
characters of the first name. When
pre-registering “placeholder” shooters (registration entries to be used for
walk-in competitors), please keep this format in mind and choose a first/last
name pair that will be uniquely identified using this convention.
10. Difficulty entering scores before time – One particularly long stage found it much faster to enter hits before time. ASS as used at the match did not make this convenient, as stats officers had to enter a “dummy time”, then scores, then the actual time. ASS has responded to my requests by adding a feature to USPSS to allow hits to be entered before time.
There were a few corrections requested by shooters:
1. One shooter reported that a “D” hit on a stage was reported as an “M” on the results. Investigation showed that a hits was “M” on the Palm and “D” on the manually completed printed score sheet. Under match policy, the score written on the signed score sheet was accepted as official and the correction was made. I believe that this was a “human error” in Palm or Paper scoring (I suspect the error was made entering data into the Palm, since shooter would probably have remembered if she had a miss on that stage. This serves to illustrate the importance of training shooters to check their Palm scores as well as their “hand-written” copy. With the exception of the previously mentioned “zero score issue”, this is the only case we encountered where the Palm score and paper score differed.
2. There were a couple of registration (not scoring) errors – Senior vs. Super Senior; a walk-in competitor whose name was correctly updated but did not have the female flag set; etc..
3. One shooter reported a zero score on a stage which he did not zero The paper score sheet was located and the correction made. I believe this was due to issue 2 described under “issues encountered.”
4. One “operator error” in manual entry of scores from a handwritten score sheet.
In reviewing the paper score sheets, I found a couple of manual errors:
1. Two
score sheets had a data entry error (due to the “auto advance” after entering
two digits default in EzWinScore – I’ve always hated that feature). Lesson learned: When scoring a match with
the Palm, any paper score sheets must be placed in a separate pile after
processing to facilitate verification of manual data entry.
2. One shooter who was “paper scored” had two paper score sheets – one marked “re shoot”. The incorrect paper score sheet was entered. The range staff should have torn up the original when issuing the re shoot - this was range officer error.
3. I manually verified one stage by entering all scores for that stage from paper into a separate copy of EzWinScore and comparing those stage results with the ones from the match results. I found a couple of times which were off by a single digit. A check with the Palms confirmed that the data in EzWinScore matched that in the Palms (so it was not a transfer error). I believe that this was an error in copying scores from Palms to Paper. Since “paper scores” are the official match document, those values were used.
4. There were a handful of registration errors (Someone not Senior, someone Senior, etc.)
1. Palm scoring can work very well, however, at this point in time paper backup of the scores is a necessary and appropriate precaution. It also provides a mechanism for shooters to “sign and accept” scores consistent with current rules.
2. The “back room” process requires a staff familiar with operation of the system. At the current state of the transfer program, I recommend Palm scoring at major matches recruit a tech weenie to run the back room.
3. “Field/Range support” (RO’s rushing to stages) is more of a stats burden than merging the scores. We were able to handle A7 with two stats people, but one could have done the job if our range crews were experienced.
4. Use screen protectors on the Palms and be prepared to replace them after a big match. RO’s can be brutal on these units.
5. We had zero Palm failures. Even so, backing up individual palms by beaming to a “Master Palm” on a regular basis is a reasonable and simple precaution.
6. A match needs either an extended posting period (5 days in the case of A7), or a final awards ceremony format which means that all shooters will be around to verify their scores on the wailing wall. This comment applies to all matches, not just “Palm specific” ones.
7. Any score sheets completed by hand must be segregated into a separate batch upon pickup, so that manual data entry (if there is any) may be manually validated.
8. The “load and make ready” command should not wait on the completion of “Palm paperwork” from the previous shooter.
9. Shooters and Range Officers must be trained to verify that the shooter name in the upper left corner of the Palm display is the correct one for the shooter (see issue #8).
10. Bring a fast printer. Palm scoring gives the capability of virtually instant results, but a slow printer can undermine speedy delivery.
11. Shooters muse be given the opportunity to review their individual hits on the Palm. RO’s must be sure to offer this to the shooter, to avoid the unintentional creation of a match culture which expects shooters to blindly accept their totals. Remember, running a match is a “customer service” operation.
12. Range Officers should be trained on how to call up a shooter’s score from a previously shot stage.
The following message posted to the unofficial IPSC mailing list by Area 8 Director George Jones is reprinted with permission. I have not included numerous other evaluations and comments from persons not present at the match.
Message: 4
Date: Sun, 31 Aug 2003 13:31:59 -0400
From: "George Jones" <hlcptr@earthlink.net>
Subject: Re: [Ipsc] Palm sscoring
To: "ipsclist" <ipsc@ipsclist.org>
Message-ID: <000b01c36fe5$ff0cd1d0$0202a8c0@Dad>
Content-Type: text/plain; charset="iso-8859-1"
----- Original Message -----
From: <catsfarm@interl.net>
Sent: Sunday, August 31, 2003 12:17 PM
Subject: Re: [Ipsc] Palm sscoring
> I have to agree totally here. It's the shooters responsibility to
verify
> the score sheet. I do the same thing.....I won't sign a score sheet
until
> the shots have been added up, and check the "time" and make sure
that
> everything is correct.
Let me start by saying that I am basically quite open-minded to technology,
so the long-term potential of Palm Scoring does intrigue me. As a small
club MD, I am also interested as to its possible application (cost not
withstanding) in my matches.
After having shot the A7 match yesterday, I'd like to provide some
impressions from the perspective of the shooter. We shot the match in the
afternoon, under partly cloudy, occasionally sunny, skies. I mention that
to give an impression concerning the ease/difficulty of viewing the screen.
Once the actual target scoring was finished, the Palm operator called out
the Hit Totals (A-x, B-x, C-x, etc) and Penalties to the RO with the paper
scoresheet, on which *only those called totals* were entered. The shooter
was then asked to initial that scoresheet (the individual target line blocks
were routinely crossed out on each stage) and was provided with the yellow
copy.
By the time that process was completed, the Palm operator was already
involved in prepping the Palm for the next shooter, so the shooter who had
just shot had no opportunity to review either the detailed hit entries (such
as we are used to seeing on a paper scoresheet), or even the total hits
report as calculated by the Palm.
So what we have, in actual practice at this (test) match, was a process
which basically requires the shooter to accept and sign data which has not
been presented in the familiar format we are all used to seeing. Not to
imply that any errors were suspected during the match, simply to say that we
no longer have the opportunity to review the total hits/misses for each
individual target for the stage. A fair amount of trust is required by
the
shooter in the accuracy of that data "transfer" between the Palm
operator
and the scoresheet RO.
In order to have the opportunity to review the details of the scoring, the
Palm operator would have to pause for that review, delaying the start of the
next shooter. I was interested in seeing it, but I did not want to do
delay
the next shooter, so I waited for a stage where l was the last shooter to
ask to see the details. Unfortunately, the Palm operator was not able to
call up the appropriate page(s), so I thanked him and moved on to catch up
with my squad. I suspect that the general lack of familiarity was at play
here and that this might improve in the future.
Lastly, while attempting the above, I found it somewhat awkward to see the
screen while the operator was working the unit. The viewing angles makes
it
difficult for the "second person" to see the display past the glare.
Since
I am not familiar with the system, I didn't even know "where to
look", so
the glare (and the fine print) made it especially difficult. I only
mention
this as a reason why reviewing the detailed info might further delay the
next shooter. Again, better familiarity might have helped, although I
don't
think it would be realistic to expect all shooters to be proficient, or even
familiar with the hardware display.
After we were finished shooting and were reviewing the printed interim
results from the morning squads, we noticed that a shooter who was not
shooting until Sunday was shown as winning a stage. I pointed it out to
Rob, who later reported that an entry error in a shooter number (which is
the primary shooter identification for the Palm) was the culprit. I am
confident that the error would have ultimately been discovered and corrected
even without our "early warning". I will leave it to Rob to
give the
pertinent details.
All in all, I remain open-minded at this point, but I was somewhat uneasy
over the inability to see my hits as accustomed. Perhaps the
(understandably) precautionary addition of the paper back-up in this match
was the cause. Had that not been in place, the shooter would have had to
"sign" the Palm results, after having reviewed those results
directly. I
can see where that might slow down the transition to the next shooter in
line somewhat, but not a deal breaker by any means.
HTH
George