TxDOT working to fix double charge problems in Central Texas Turnpike, filtering them out meanwhile

A misaligned open road toll antenna on SH45 throwing up duplicate transponder and video tolls in large numbers in July has led indirectly to discovery of lower incidence problems throughout the Central Texas toll system, and also in neighboring systems with which TxDOT does business. The wrongly pointed transponder antenna meant that some trips under the antenna were not being tolled at all while those in a nearby lane, and sometimes ahead or behind, were being tolled by transponder and by camera (video toll) as well - generating a double toll.

TxDOT is refunding all the double tolls it can find.

915MHz frequency signals are focussed from the gantry above the road into a cone shaped zone that covers a roughly oval area of roadway. In an open road situation the cones of coverage cover the whole of the roadway including shoulders with roughly one cone per lane. They can get skewed by a knock, high winds or vibration, or just because they weren't set correctly to start with.

Director of toll operations and IT at the Turnpike Division of TxDOT, David Powell told us today: "Lake Creek Plaza was generating a completely unacceptable level of false charges, but it was obvious once we looked and quite easy to fix. Other problems we've found, at quite low levels are more elusive. We're installing (software) filters to prevent these being posted in future and we're going back (through the files) and crediting accounts where we find double charges."

"We've opened six roads in six months" Powell said, suggesting they were most focussed on getting systems up and running, and only recently have had time to search out problems.

Powell said they tested the toll systems before opening, but low incidence deficiencies aren't thrown up in that kind of testing because the number of tests is too low to draw attention to them.

Two or three thousand passes is regarded as thorough testing but a one in a thousand problem may not show up at all or produce such a tiny handful of errors that their source can't be tracked down. Only real traffic volumes with tens of thousands of passes per day over weeks produce the numbers of transactions needed to identify low incidence problems.

Gabriella Garcia, a TxDOT spokesman says the present estimate is that 50k of 32m or 0.156 percent of transactions, one in 640, involved double charges since the Central Texas tollroads began tolling at the beginning of the year. She said she had no estimates of the numbers at Lake Creek Plaza.

Powell told us the initial Lake Creek Plaza problem was rather easily found and fixed once they saw the incidence of double tolls there. It took customer complaints to catch their attention. But a wider system problem emerged in examining transaction files on Lake Creek Plaza – a low incidence of double charges all over the Central Texas Turnpike and at some of the associated tollroads run by CTRMA and even HCTRA. This probably has multiple causes – smaller antenna misalignments, antenna power settings, and timing issues.

There's a Microsoft DLL (CORRECTION MADE HERE - we erroneously reported DDL before) or subroutine in commonly used database systems including those of the toll management system that causes periodic timing mistakes, Powell said, that could be to blame in some cases.

[An amusing account of DLL problems is found here http://en.wikipedia.org/wiki/DLL_hell]

Since tracking down low incidence problems takes time – Powell calls it "searching for a needle in a haystack" – they are installing new filters for double charges. The filters identify charges very close in time and filter them out so only one of two charges is actually posted to the customer's account.

COMMENT: A bit surprising the toll management system didn't have this kind of filter built-in from the start. But this is a relatively small glitch in an otherwise successful strartup. Traffic is great.

UPDATE: TxDOT say their system has now been improved to the point where it is generating about 50 "duplicates" or potential double-charges a day to be credited back to customer accounts. That is 0.03% of transactions or about one in 3,300.

The filter to automatically prevent these remaining double-charges will be operating "soon," they say.

Kai Chen, a reader, writes us on DLL problems: "Dynamic Link Libraries (DLL) can reside either at the application (database software, per your report) or in the Operating System.  Depending on its source, it may not even come from Microsoft.
Fixes are difficult to come by as the vendor needs to replicate the issue in order to fix it.  If it is an operating system DLL (Microsoft DLL) it is already difficult to get their attention. Timing issues are notorious(ly difficult) to duplicate so I doubt a fix is in store for the short term." 

Meanwhile that leaves it to the filter to prevent the double charges. TRnews. 

TOLLROADSnews 2007-10-15 UPDATES 2007-10-16 1800, 2007-10-18 1025