Across the globe, casinos continue to move away from analog systems in favor of IP-based surveillance solutions. Integrations that enable video, alarm and transactional data overlay to achieve unprecedented levels of situational awareness are becoming the norm, not the exception.
“Probably the most important thing for gaming operators to pay attention to is whether their provider has listened to them and truly understood their specific requirements in terms of regulation compliance, operational and technology needs. It’s only by identifying the key needs of a casino that worst case scenario planning is effective, and that’s what has to happen to maintain system resiliency.”Adam Heisler, Sales Director for International Gaming, Synectics
Data has never been more useful to the gaming industry; it has also never been more important to protect. In this interview, we talk to Adam Heisler - Synectics’ Sales Director for International Gaming - about system resiliency and failover solutions, and learn how with the right system and camera technology nothing has to be left to chance when it comes to safeguarding surveillance.
Why is surveillance resiliency so important for the gaming industry?
In gaming, data loss can have massive financial ramifications. Even a few minutes of lost video can incur significant regulatory penalties – in some Asian jurisdictions fines can be as much as up to 10% of the casino’s yearly gross earnings. In other regions video coverage loss may force an entire casino to be shut down resulting in loss of revenue for every minute of closure. This is the case in Ontario, Canada for example.
Setting aside monetary penalties associated with regulation, the operational implications of system failure are also significant. For example, a network failure in an all IP surveillance system could result in a catastrophic loss of video coverage that exposes casino operators to threats such as theft, fraud, security breaches and (in an increasingly litigious world) injury liability.
What types of failover should operators expect from their surveillance solution?
Because of their scale and the volume of data most casinos deal with on a daily basis, almost all solutions for the gaming industry now offer RAID-6 redundancy as standard. Unlike RAID-5, RAID-6 is able to protect against two simultaneous disk drive failures by writing two independent parity stripes across all drives.
Whether operating an analog, IP or hybrid system, gaming operators will also benefit from what we call “hot swap” technology to guard against hardware failure. It offers the ability to automatically relocate recording in the event of, for example, an encoder or storage server malfunction.
With an integrated security management solution such as Synectics’ Synergy 3, the location of recorded footage is intuitive – the best location for ‘back up’ is identified and utilized with all failover actions tracked by the system to ensure video is managed at all times to guarantee seamless viewing and playback. It’s a multi-layered approach that removes any single point of failure. It’s our recommendation that any hot swap server needs to be minimally as big as the largest server.
It’s also worth pointing out that system or equipment failure is not the only reason for building in multiple levels of failover. Hot swap functionality is also useful for activities like scheduled system maintenance. Operators or technicians are able to instruct the system, for instance, to divert footage during a set time frame on a specific day.
What about edge recording? Is that something that gaming properties need to consider?
It certainly is. Edge recording (sometimes called ‘removable on-board storage’) refers to saving data directly to the camera, or storage capable encoder (built-in SSD).
Most of the redundancy measures I’ve talked about so far will provide all the ‘fail safe’ protection that a property will ever need. BUT, there are always exceptions to every rule. For example, if a network switch dies or a camera connection is severed, then access to centralized storage and redundancy is cut. Without the facility to record locally to the device, that data would be lost. Edge recording ensures this isn’t the case.
What about when the connection is restored? Doesn’t that present a problem in terms of returning to ‘normal’ recording?
Not if the system is designed correctly and the right management software is in place – in fact it is a simple, intuitive process. A good video surveillance system will ‘know’ what to do.
With Synergy 3, for example, if a network failure occurs, edge recording to the “local device” (e.g. camera SD card) is automatically triggered (if not set as default to “always record”) and continues recording until the connection issue is resolved. Upon resolution of the network connection, the software automatically retrieves the data stored on the SD card and restores it to the normal storage array via the newly reconnected network – we call this Backfilling. When viewed later by an operator, it appears as seamless, continuous footage.
One thing that operators need to be sure to ask their system provider about, however, is how edge recording and backfilling impacts on live viewing. With the right cameras and management software, viewing, recording and backfilling should be able to take place simultaneously without affecting image quality or speed. Some cameras and encoders do not have this capability and is something that should be checked before being put into use.
This is hugely important. The casino surveillance team need to know they can view live footage - without image latency - to track ongoing situations, even as recording/backfilling is taking place.
Would a casino ever purely rely on edge recording and backfilling as its failover solution?
We wouldn’t recommend it. Backfilling can save the day (and often has!) but it also has its drawbacks. It should be viewed as an add-on to failover solutions, not a stand-alone option.
The most obvious issue is the potential for camera damage (deliberate or accidental). If footage is only being recorded local to the camera – to internal memory or SD cards – then damage here means data loss.
Capacity is the other significant issue. While ideal as a ‘last line of defense’ should an incident occur, recording large volumes of data, particularly HD quality at full frame rate, is not a role suited to on-board storage.
It’s also not very practical to have this facility on all cameras, especially when you consider that a typical SD card will need replacing around every 6-8 months if being recorded to at all times. Some of the properties we work with have thousands of cameras. It simply wouldn’t make sense to equip all these for backfilling from a cost and maintenance standpoint.
"With the right solution, casinos can actually prioritize individual cameras, to ensure that storage capacity is allocated to the most vital footage."
Are there any other practical factors gaming operators need to consider in terms of failover and redundancy?
There is another interesting point to discuss regarding storage capacity. Imagine there is a catastrophic error which effectively takes out ‘normal’ recording for a whole rack of 250 cameras. As I’ve already highlighted, recording can be automatically diverted to the best failover solution available – something all good systems will achieve. But there will only ever be a finite amount of footage that can be stored. What then? Here’s where things can get really smart.
With the right solution, casinos can actually prioritize individual cameras - ranking them 1-10 effectively (1 being highest and 10 lowest) - to ensure that storage capacity is allocated to the most vital footage. While a cash cage camera may be a priority 1 (highest), a more general foyer camera may be a 6 or 7.
It’s also important to think about things like power supply. For edge recording and backfilling to take place in the event of a PoE network switch failure, an additional power supply to cameras is required. One thing to be aware of is where dual power sources are available, many cameras will automatically reboot if either power supply cuts out for any reason. In some cases this may result in downtime. Casinos should check with their surveillance solution provider to ensure this won’t happen.
Are there any other checks you’d recommend?
Probably the most important thing for gaming operators to pay attention to is whether their provider has listened to them and truly understood their specific requirements in terms of regulation compliance, operational and technology needs. It’s only by identifying the key needs of a casino that worst case scenario planning is effective, and that’s what has to happen to maintain system resiliency.
Any thorough surveillance provider will develop and test their proposed solution under the most extreme conditions – scenarios that would rarely happen but if they did, would generate the most stress on the technology in place. If the casino’s data priorities can still be met in these exceptional circumstances, then that’s the right solution.