The Defense Personal Property System (DPS) is USTRANSCOM’s logistics system that moves shipments of personal goods of more than 1.3 million service members, 700,000 civilians, and their families. However, DPS crashed after a March 10, 2017, production update. After the crash, the site began experiencing unacceptable standards of performance, with latency of over five minutes, which the Defense Digital Service (DDS) team considers comparable to a non-functioning site. The poor usability and functionality of DPS is a major stress factor for military and civilian families who are in the midst of frequent relocations across the globe, and contributes to numerous failures and delays in delivering and receiving family household goods.
DDS deployed a Rapid Response Team (RRT) to provide immediate remediation to DPS, and and rescued the site within four days. After DDS’ RRT quick remediation, DPS users were reporting significantly reduced latency rates with more than 4,500 concurrent users — the largest amount of concurrent users DPS has ever supported. Most importantly, shipment requests that were successfully requested and scheduled increased from 16% to over 99%.
During the recovery of DPS, DDS helped USTRANSCOM to optimize their database and took steps to ensure that the traffic from automated computer scripts written by third parties would not impact the system performance for service members and their families. In order to prevent traffic from these scripts from consuming all available server resources, DDS helped staff implement two tools capable of throttling automated script traffic, as well as the ability to blacklist specific IP addresses being used by overly aggressive bots. These steps better distributed load to the system, freeing up servers to handle more overall traffic, resulting in increased speed for users. Early metrics indicate that third party services are adapting their behavior to the new system rules, and DPS has been significantly more stable since March 27, 2017, just two weeks after the crash.