Maximum Security:
Digital ComBridge Has Prison Phone Systems All Locked Up

Rob Comstock and Jay Hogg, T-NETIX

A hush descends over the courtroom as the judge asks the jury if they've reached a verdict. "Yes," they say and the prisoner rises to hear the decision. The foreman pronounces the accused guilty and the court explodes with commotion. The prisoner is led away, screaming threats to the jury, the judge, and witnesses. An episode of TV's Law & Order? Maybe.

But in many cases this scene comes close to reflecting reality. Using the telephone as a primary means of communication, prisoners have been discovered making threats, running businesses - even engaging in further crimes while imprisoned. Key to preventing this type of activity is controlling prisoner access to the public telephone network. Traditionally, the only tool available was ‘ 0+’ collect-call dialing from telco service providers, which was a partial solution at best. For instance, calls could not be blocked to specific numbers from specific prisoners, recorded for crime prevention purposes, or metered for fraud prevention. Clearly, prison administrators needed better tools.

Telephone lockdown

Enter our company's product, Digital ComBridge. Running QNX 4, Digital ComBridge completely blocks a prisoner's direct access to the public telephone network. Each inmate is provided with a unique personal identification number (PIN). Access can be denied to specific numbers on an inmate-by-inmate basis, by time of day, and by type of call. Before completing a connection to an authorized number, an interactive voice response system (IVR) informs the called party of the name of the prisoner and presents them with options: accept the call, reject it, or permanently block further calls from that prisoner. Because many calls are collect, frequently called parties, such as family members, can specify the maximum monthly charges they are willing to accept. The system will then provide warnings as the charges approach their limit. Once the limit is reached, the system will block calls to that number.

In addition to preventing unauthorized calls, Digital ComBridge incorporates surveillance features to help law-enforcement officers uncover criminal activities. For instance, all call details are stored in a Sybase database from which officials can run reports to discover unusual calling patterns, such as more than one prisoner calling the same number. Armed with this information, they can then listen to call recordings made by Digital ComBridge to determine if further investigation is required. These features are a boon in fighting organized crime and are often used by the FBI.

Making the break from DOS

Introduced in 1997, the system is now running in over 1600 correctional facilities with over 40,000 telephone lines, making T-NETIX the leading supplier of telephony security and monitoring systems to the correctional services sector. In fact, our company holds a 30 percent market share. The standard configuration includes an ISO-based rack-mounted chassis with a single Pentium 166 processor, 32M of RAM, and 11 to 19 slots for telephony cards such as ISDN and T1.

We developed Digital ComBridge to replace an older DOS-based application that had reached the end of its useful life. We needed to replace this system with one that could handle more lines; provide better networking, scalability, and multi-node management; and allow us to run 24x7 - which meant fault tolerance and the ability to do live upgrades. We also required an OS that would be lightweight and highly customizable, so that system resources could be dedicated to applications - where they belong. In addition to these essential requirements, we had a long list of goals we wanted to achieve. For instance, since our company is a service provider as well as a vendor, we already had a large investment in systems running 386 and 486 processors that we wanted to retain. We also wanted the option of producing an embedded, flash-based version of the system.

Evaluation process

With these goals in mind, we set out to find an operating system to meet our needs. Initially, we evaluated OS/2, Windows NT, Linux, other Unixes, VxWorks, and pSOS, and spent several months developing prototypes to evaluate each. However, none of these operating systems met our requirements. OS/2 and Windows NT, for instance, clearly would not run on our legacy equipment. Nor did they support the revision control that was required to properly upgrade and support field installations. While the Linux and Unix systems offered POSIX compliance, they didn't allow the live upgrades we needed for 24x7 uptime.

Finally, serendipity stepped in to lend a hand. In November 1995, some members of the T-NETIX team were in San Francisco. Purely by coincidence we shared a hotel elevator with employees of QNX Software. We struck up a conversation and asked for a copy of the operating system to evaluate. We began our evaluation of QNX 4 in December and had a working prototype of the system up and running in two weeks! By January we'd decided on QNX and begun development of Digital ComBridge in earnest.

The right choice

QNX was chosen because it excelled in all the areas that are key to making this project successful. The fact that QNX is used in nuclear power plants, assembly lines, telecommunications, postal sorting, DNA fingerprinting, the space shuttle, and major communication networks like Compuserve and Visa International definitely helped us make our decision. We were also impressed by the accessibility of QNX support and development personnel. Depending on the nature of the difficulty, resolutions were often received in as little as 30 minutes and sometimes included not only instructions but a program that fixed a problem. We were also impressed by the receptiveness of QNX to observations and suggestions. The development team, used to working with DOS, was easily sold on the benefits of working with QNX the company as well as QNX the OS.

Another key factor was QNX's modular design and true memory management unit (MMU) protection for the microkernel. This design offered not only greater reliability but a number of other benefits. For example, the system can be easily customized to the requirements of each site. The send/receive/ reply (SRR) architecture offered by QNX worked perfectly to allow us to design custom installations with just the pieces needed. Leveraging this architecture, we were able to greatly reduce overall development costs by reusing modules in ways that are tailored to the needs of our customers.

Room for change

The modular QNX architecture also aided us in dealing with the quickly changing nature of corrections-related technology. We needed to design interfaces to our applications that were generic enough to accommodate changes in third-party technology while remaining powerful enough to perform reliably in a large, distributed network. We achieved this by exposing global names of the processes that communicate with interfaces. All nodes in the switch environment can access these processes, and the process itself implements the protocol and has access to the methods needed, be they serial, TCP/IP, client database, or some other, to communicate with the new technology. Because the QNX microkernel is built with a philosophy that simplicity is a virtue - and avoids the one-size-fits-all design approach that results in so many bloated operating systems - this approach allowed us to put all the horsepower behind the applications, not the OS. The strategy has been so successful that we currently have systems with 72 phones, 72 outbound lines, 72 DSP channels, recording, database, remote access, and TCP/IP networking on 486/100-based PCs with 32M of memory. System utilization peaks at around 18M and 50 percent processor utilization on this configuration with all lines in use.

With QNX, we were also able to meet the goal of upgrading software modules on live systems to support configuration changes and on-the-fly maintenance. And we had to deal with the crucial concern that some changes in telephony application configurations can create logical paradoxes, which typically require a great deal of coding to deal with. For example, we had to anticipate changing the maximum call duration on a system from 30 minutes to 15 even when a call was currently in progress at 22 minutes. We achieved these goals by loading configurations into named, shared memory regions, which programs then map in. Using this method we can ‘ delete’ and ‘ reload’ a configuration into memory and all currently running programs will keep the old configuration until they have exited. New instances of programs will use the new configuration.

When our development project was complete we weren't disappointed that we'd chosen QNX. The first installation of Digital ComBridge ran for over 18 months without a single problem - longer than the 16 months it had taken to develop! And it would have continued running had it not eventually required a memory upgrade.

System maintenance

Because the Digital ComBridge software is installed in literally thousands of sites around the world, system maintenance is a critical issue. Not only do we need to be able to manage versioning and regular maintenance tasks, it's also critical that we be able to access sites remotely for realtime troubleshooting, in the event that problems arise at a customer location. QNX helped us meet these requirements in a number of ways. Robust support for TCP/IP greatly simplifies remote installation and upgrade procedures, and makes reliable connectivity for troubleshooting a snap. Troubleshooting and maintenance are also facilitated by QNX. Support personnel can use the industrial-strength TCP/IP support to telnet into any site in the world and fix problems in real time. The reliability and low bandwidth requirements of a text-based interface were crucial for this purpose. Other systems such as NT, which require a graphical interface even on remote connections, impose a substantial bandwidth penalty. This could prove to be a fatal flaw if support personnel couldn't connect to a site because network problems were making access difficult. Using telnet, even low-speed direct modem connections provide useful connectivity.

Simple site installs

Performing site installations involves running a QNX boot diskette that loads TCP/IP and then mounts an exported installation tree from an NFS-based installation server. An installation script then runs on the server and prompts the user with a series of questions before installing the complete Digital ComBridge system, including initial configurations and licenses. When the system is restarted, it can place calls and only needs final site-specific configuration changes. This approach not only reduced the initial software installation time from over an hour to around five minutes, it also provides our support personnel with centralized control over all software being deployed, and eliminates the risk of support complications due to old copies of the software floating around on diskettes. It also makes it easy to build customized configurations for major installations.

Digital ComBridge also makes extensive use of QNX's FLEET networking protocol to form a switching platform by scaling to as many systems as are required in a software-transparent manner. Using FLEET, we have the scalability we need to accommodate rapidly growing demand. The ability to install additional interfaces between nodes and combine them into one ultra-high bandwidth pipe ensures that applications will continue to have the connectivity they require. The built-in load balancing and dynamic configuration of FLEET makes it virtually ‘plug and play’ to install additional nodes as they are needed.

Already, the County of Los Angeles, the largest county in the US, runs a 19-node site with 2500 lines, 1250 phones, and 1200 channels of digital signal processing (DSP). FLEET helps ensure that we will be able to keep up with the rapid growth of this and other sites.

With Digital ComBridge on guard in LA County and around the world, judges, juries, and the victims of crime can all sleep a lot easier.

For more information about Digital ComBridge from T-NETIX, visit www.t-netix.com or call 1 800 531-4245.