Telstra’s massive nerve centre exposed

The GOC (as Telstra like to call it) is an impressively enormous facility which Telstra says is continuously upgraded to monitor Telstra’s entire network across Australia and overseas.

To get some idea of the magnitude of the scale of monitoring required in this massive facility: the Telstra network handles almost 5 billion PSTN calls per year 9 billion SMS messages 10 million Telstra mobile services and over one million wireless broadband customers. The center monitors the whole core NextIP network (the IP-based backbone of all of Telstra’s networks) GSM Next G PSTN ADSL and cable networks.

The GOC is staffed 24/7 with dedicated teams looking after Telstra’s various operations including monitoring network performance the Next G mobile network Bigpond Broadband Wireless Broadband and broadcasting of content to Next G mobiles. The digital video wall measuring 24 x 2.5 metres provides an overview of the entire network along with live news feeds and weather information.

The video wall provides real time colour-coded status of the network with green indicating the network is performing adequately yellow indicating performance issues and red indicating problems. Teams on the floor can drill-down to find the exact location of the problem within Telstra’s network.

Group Managing Director Michael Rocca in the special viewing room. This is the guy who runs all of Telstra’s networks. Nice guy but at the same time you definitely wouldn’t want to cross him — he knows what he is talking about and would not be the kind of guy who could be fobbed off.

A view from the floor of the GOC showing staff monitoring various aspects of Telstra’s network in real time

[#PAGE-BREAK#Monitoring Next G broadcasting#]

Over the decade since the GOC opened Telstra has changed the way the network is monitored from a network-centric view where alarms were logged to a customer-centric view where performance as perceived by the customer is monitored. Numerous bots throughout the network collect performance parameters such as data transfer rates time to authenticate purchases on Bigpond latency and congestion. The screens show Next G broadcasts as they are streamed allowing staff at the GOC to adjust processing parameters to ensure the best possible picture on Next G mobiles.

The screen on the right monitors purchases of Bigpond downloads allowing the engineer to immediately become aware of any problems with content distribution. Graphs show current usage of services allowing resources to be reallocated during peak periods.

[#PAGE-BREAK#Keeping a close eye on BigPond#]

Telstra broadcasting operations (BigPond TV) has a dedicated team looking after bandwidth and processing resources to ensure that the network can cope with peaks in demand such as when there is live streaming of major sporting events.

Colour-coded screens monitor the performance of Bigpond’s server farms.

Customer-centric monitoring within the GOC allows the engineers to see exactly what Bigpond customers experience on their mobiles.

Screens on the side of the GOC allow staff to monitor real time performance of television transmission from overseas and within Australia including Foxtel.

The GOC has 24 hours of battery and UPS backup for all equipment as well as two diesel generators outside which can individually power the entire centre. Each generator has at least 7 days of fuel on site. Primus which has had several catastrophic data centre outages affecting the customers of many Australian ISPs could learn a thing or two from Telstra on this point.

A view of the GOC from outside showing a second diesel generator for backup power.

Telstra takes security seriously and has a full backup GOC “somewhere in the Melbourne CBD” as well as additional backup centres in various locations around Australia.

A Telstra mobile tower with storm clouds brewing overhead.