Safeguarding Gaming Surveillance

Data has never been more useful to the gaming industry; it has also never been more important to protect. In this blog, we discuss system resiliency and failover solutions and learn how, with the right system and camera technology, nothing has to be left to chance when safeguarding surveillance.

Why is surveillance resiliency so important for the gaming industry?

In gaming, data loss can have massive financial ramifications. Even a few minutes of lost video can incur significant regulatory penalties – in some Asian jurisdictions, fines can be up to 10% of the casino’s yearly gross earnings. In other regions, video coverage loss may force an entire casino to be shut down, resulting in a loss of revenue for every minute of closure. This is the case in Ontario, Canada for example.

Setting aside monetary penalties associated with regulation, the operational implications of system failure are also significant. For example, a network failure in an all-IP surveillance system could result in a catastrophic loss of video coverage that exposes casino operators to threats such as theft, fraud, security breaches and (in an increasingly litigious world) injury liability.

What types of failover should operators expect from their surveillance solution?

Because of their scale and the volume of data most casinos deal with daily, almost all solutions for the gaming industry now offer RAID-6 redundancy as standard. Unlike RAID-5, RAID-6 can protect against two simultaneous disk drive failures by writing two independent parity stripes across all drives.

Whether operating an analogue, IP or hybrid system, gaming operators will also benefit from what we call “hot swap” technology to guard against hardware failure. It offers the ability to automatically relocate recording in the event of, for example, an encoder or storage server malfunction.

With an integrated security management solution such as Synectics’ Synergy software, the location of recorded footage is intuitive – the best location for ‘backup’ is identified and utilised with all failover actions tracked by the system to ensure the video is managed at all times to guarantee seamless viewing and playback. It’s a multi-layered approach that removes any single point of failure. It’s our recommendation that any hot swap server needs to be minimally as big as the largest server.

It’s also worth pointing out that system or equipment failure is not the only reason for building in multiple levels of failover. Hot swap functionality is also useful for activities like scheduled system maintenance. Operators or technicians can instruct the system, for instance, to divert footage during a set time frame on a specific day.

What about edge recording? Is that something that gaming properties need to consider?

It certainly is. Edge recording (sometimes called ‘removable onboard storage’) saves data directly to the camera (built-in SSD).

Most of the redundancy measures talked about will provide all the ‘fail-safe’ protection that a property will ever need. But there are always exceptions to every rule. For example, if a network switch dies or a camera connection is severed, then access to centralised storage and redundancy is cut. That data would be lost without the facility to record locally to the device. Edge recording ensures this isn’t the case.

What about when the connection is restored? Doesn’t that present a problem in terms of returning to ‘normal’ recording?

Not if the system is designed correctly and the right management software is in place – it is a simple, intuitive process. A good video surveillance system will know what to do.

With Synergy, for example, if a network failure occurs, edge recording to the “local device” (e.g. camera SD card) is automatically triggered (if not set as default to “always record”). It continues recording until the connection issue is resolved. Upon resolution of the network connection, the software automatically retrieves the data stored on the SD card and restores it to the normal storage array via the newly reconnected network – we call this Backfilling. When viewed later by an operator, it appears as seamless, continuous footage.

Operators need to ask their system provider about how edge recording and backfilling impact live viewing. With the right cameras and management software, viewing, recording, and backfilling should occur simultaneously without affecting image quality or speed. Some cameras and encoders do not have this capability, so they should be checked before being used.

This is hugely important. The casino surveillance team needs to know they can view live footage – without image latency – to track ongoing situations, even as recording and backfilling is taking place.

Would a casino rely purely on edge recording and backfilling as its failover solution?

We wouldn’t recommend it. Backfilling can save the day (and often has!), but it also has drawbacks. It should be viewed as an add-on to failover solutions, not a stand-alone option.

The most obvious issue is the potential for camera damage (deliberate or accidental). If the footage is only being recorded locally to the camera – to internal memory or SD cards – then damage here means data loss.

Capacity is the other significant issue. While ideal as a last line of defence should an incident occur, recording large volumes of data, particularly HD quality at full frame rate, is not a role suited to onboard storage.

It’s also not very practical to have this facility on all cameras, especially when you consider that a typical SD card will need replacing around every six to eight months if being recorded at all times. Some of the properties we work with have thousands of cameras. It simply wouldn’t make sense to equip all these for backfilling from a cost and maintenance standpoint.

"With the right solution, casinos can actually prioritise individual cameras, to ensure that storage capacity is allocated to the most vital footage."

Are there other practical factors gaming operators need to consider regarding failover and redundancy?

There is another interesting point to discuss regarding storage capacity. Imagine there is a catastrophic error which effectively takes out ‘normal’ recording for a whole rack of 250 cameras. As already highlighted, recording can be automatically diverted to the best failover solution available – something all good systems will achieve. But there will only ever be a finite amount of footage that can be stored. What then? Here’s where things can get smart.

With the right solution, casinos can prioritise individual cameras – ranking them 1-10 effectively (1 being highest and 10 lowest) – to ensure that storage capacity is allocated to the most vital footage. While a cash cage camera may be a priority 1 (highest), a more general foyer camera may be a 6 or 7.

It’s also important to think about things like the power supply. For edge recording and backfilling to take place in the event of a PoE network switch failure, an additional power supply to cameras is required. One thing to be aware of is where dual power sources are available. Many cameras will automatically reboot if either power supply cuts out for any reason. In some cases, this may result in downtime. Casinos should check with their surveillance solution provider to ensure this won’t happen.

Are there any other checks to recommend?

Probably the most important thing for gaming operators to pay attention to is whether their provider has listened to them and truly understood their specific requirements regarding regulation compliance and operational and technology needs. It’s only by identifying the key needs of a casino that worst-case scenario planning is effective, and that’s what has to happen to maintain system resiliency.

Any thorough surveillance provider will develop and test their proposed solution under the most extreme conditions – scenarios that would rarely happen but, if they did, would generate the most stress on the technology in place. If the casino’s data priorities can still be met in these exceptional circumstances, then that’s the right solution.