Viakoo OnPremises Best Practices Guide

ViakooAppButtonFinal199x199.jpg

Best Practices for Viakoo On-Premises Infrastructure

 

Introduction

A proper infrastructure is a must in order for Viakoo On-Premises (VoP) to work properly. Viakoo is an enterprise-class product designed to operate as a cluster of virtual machines within an enterprise-class cloud, either a vSphere / vCenter cloud with enterprise-class storage, or Amazon AWS with enterprise-class storage. It was not designed as a desktop or server product to run on consumer-grade equipment.

 

The purpose of this document is to give you guidelines for how to build the enterprise-class infrastructure needed to successfully implement Viakoo within your company, as well as best practices for design and management of the private cloud needed to run Viakoo. It is divided into multiple categories based upon the size of your infrastructure.

Overview

The Viakoo solution is a cloud-based solution for monitoring, visualizing, and diagnosing your video surveillance infrastructure. The solution consists of three components:

 

  1. Recorder agents -- these are small programs that live on your video recorders and report to:
  2. Collector agents -- there is one of of these per (physical or virtual) site, often also installed on one of the video recorders. Collector agents must be able to reach via networking both the recorder agents and the next component:
  3. Viakoo Service:  This is a set of virtual machines running in an enterprise cloud environment that accepts the data from the collector agents, stores it in an enterprise class database, runs a reasoning engine against the state of a site to determine whether there are anomalies, tickets any anomalies that it finds, and otherwise allows visualizing, alerting, and diagnosing your video surveillance network and your video recorders. 

The Viakoo Service itself can run in one of three different places:

  1. Public Cloud: This shares hosting with other Viakoo customers. If you have fewer than 10,000 cameras total, we recommend this as the most cost effective solution. We handle all IT necessary to provision, maintain and update the general public cloud solution. We can provide dedicated IP addresses to allow your Collector Agents to talk to our public cloud solution.  Sites reporting to the public cloud solution need a relatively trivial amount of outgoing data bandwidth -- approximately 1k bytes per second per 1,000 cameras -- and the Viakoo collector agents cache data if the service is not reachable and retry until the service is reachable so even unreliable connections are suitable for use with the general public cloud. This document does not further describe best practices for this solution.
  2. Dedicated virtual private cloud. If you have more than 10,000 cameras total, we recommend this rather than Viakoo OnPremise as the most cost effective solution. We provide you with your own virtual private cloud implemented in our data centers and handle all IT needed to provision, maintain, and update the dedicated virtual private cloud solution. No other Viakoo customer has access to your private cloud’s servers or data, and all data at rest is encrypted so that our data center provider or other entities cannot access it. If your country’s privacy policies do not allow locating data in the United States, this dedicated private cloud can also be provided at extra cost in a data center located in a geographical area closer to you such as in Europe, West Asia, or East Asia. All data between sites and your private cloud are sent encrypted (via SSL), and all communications between your web browser and your private cloud occur via encrypted HTTPS. This document does not further describe best practices for this solution.
  3. Viakoo OnPremises: You provide the equipment and provision the VMware VSphere-based private cloud and enterprise-class storage subsystem needed to implement the Viakoo Service at your central office. This requires extensive IT knowledge on your part as well as a significant up-front investment in terms of time and money. It also requires that Viakoo personnel be allowed to access your Viakoo OnPremise solution a minimum of twice per year in order to maintain and update it as necessary. Without this regular maintenance your Viakoo OnPremise installation will eventually fail. This is the most expensive solution both in terms of your time and money, and in terms of Viakoo support requirements, and should be undertaken only if you have a significant number of cameras (thousands to tens of thousands) and have regulatory requirements that prohibit exporting data to outside your company or country even to a dedicated private cloud. This document describes best practices for the OnPremise solution.

General best practices

  1. Viakoo Service was designed as an enterprise service to scale up to 2,00,000 cameras in a single constellation, it is not cost effective to scale down to 500 cameras for an OnPremises implementation. Ideally, VoP should  be implemented at a central office location that all branch offices can communicate with, not at branch office locations. Only the Viakoo Agent should be installed at branch office locations.
  2. Always overspecify your equipment. VIakoo is continually adding functionality to the product, and new functionality often needs additional resources.
  3. For all but the very smallest of installations,  storage and compute should be separate resources. This allows adding additional storage and compute resources as needed and allows replacing a compute resource without losing data (a different ESXi server can be pointed at the storage and told to run the virtual machines on that storage, either automatically via vCenter, or manually via the vSphere GUI).
  4. If you insist upon running storage locally on a VMware ESXi box, you need a RAID controller with a large RAM cache and battery backup. Otherwise writes to the storage will run extremely slowly, to the point of being useless for Viakoo.
  5. Viakoo’s database is write-intensive (approximately 50% writes, 50% reads) and optimized for SSD’s. Only smaller sites with a few hundred cameras will run adequately fast with rotational storage.
  6. Partitions on SSD drives must be aligned on 8K block boundaries for maximum performance. Filesystems located on SSD drives exported to VMware as storage must be aligned on 8K block boundaries for maximum performance.
  7. Disks or SSD’s used for Viakoo’s database should be RAID’ed with RAID10 or equivalent, not RAID5 or RAID6. RAID5 and RAID6 cause write amplification, which along with the natural write amplification inside SSD devices (they typically have erase zones of 16mb or more) causes drastic and dramatic performance slowdowns, often to the point of being slower than rotational storage.
  8. For maximum performance, Viakoo “sites” should be kept at 500 cameras or fewer. Viakoo will operate with sites of up to 1,000 cameras but slows down drastically after 500 cameras due to the way the reasoning engine inside the Viakoo Service looks at the global state of sites in order to reach conclusions about the state of a site. Large installations with more than 1,000 cameras can be divided into multiple virtual “sites” splitting the video recorders and cameras between them in order to speed processing.
  9. The Viakoo Service virtual machines are already instrumented to be monitored by Nagios. A list of the Nagios sensors available can be provided upon request, as well as example Nagios configuration files for monitoring an example cluster that can be integrated into your own Nagios installation.

Tiny Installations (under 1000 cameras)

Introduction

It is recommended that you install the Viakoo service at a centralized location within your infrastructure and have multiple small sites report to that centralized location, rather than have individual Viakoo service installations at small sites . For example, if you are a bank, it is recommended that Viakoo be installed at your central office and the collector agent at each branch office report its status back to the central office via your secure network. This is because there is significant overhead running the Viakoo server both in terms of manpower, maintenance, and equipment, and replicating this overhead at each branch office is a significant expense. By contrast, the collector agent that runs on branch offices was designed to be a lightweight product that installs on one of your video recorder servers and requires no maintenance once installed.

 

In some cases, however, due to lack of secure or reliable networking it may be necessary to install Viakoo at the branch office. Or the branch offices have so few cameras that there are less than 1,000 cameras total once all collector agents report to the central office. These are our recommendations:

Stand-alone Server specifications

  • Operating system: ESXi 6.0 or later
  • ESXi-compatible motherboard and chassis. See: https://www.vmware.com/resources/compatibility/search.php
  • Minimum number of processors: 1
  • Minimum number of cores: 12
  • Minimum amount of memory: 32GB
  • RAID controller: Recommended: Broadcom MegaRAID SAS 9380-4I4E Raid controller with CacheVault Flash Module (CVFM04), CacheVault Power Module (CVPM02), or better.
  • Chassis and hard drives: minimum 2U 12-disk (3.5”) storage server chassis with twelve HGST Ultrastar 15K600 HUS156060VLS600 (0B23663) 600GB 15000 RPM 64MB Cache SAS 6Gb/s 3.5" Enterprise Hard Drive Bare Drive (or better) formatted as RAID10 (3.6tb effective storage).
  • *OR*: minimum 1U 10-disk (2.5”) storage server chassis with ten Crucial MX300 2.5" 525GB SATA III 3-D Vertical Internal Solid State Drive (SSD) CT525MX300SSD1 SSD drives (or better) formatted as RAID10 (2.6tb effective)

 

This server should be attached to the video surveillance network with a virtual network preconfigured. The Viakoo virtual appliance .ova file should then be downloaded to a Windows system with access to this server for installation.

Using your existing VMware private cloud

  • Must have access to a minimum of 2.6tb of enterprise-class storage, either SAS or SSD.
  • Video recorder network must be defined via VMware’s Software Defined Networking and physically connected to your video surveillance recorder network.
  • vSphere account used to provision the virtual machines must have access to the video surveillance network plus must be able to provision the storage needed for the database instance.
  • Four virtual machines are created as part of the installation: GUI (1gb memory, 2 CPU cores), API (16gb memory, 4 CPU cores),  DBMS (8gb memory, 4 CPU cores), and PuppetMaster (1gb memory, 2 CPU cores). The CPU cores should be 2.4ghz or better. You must have sufficient resources in your VMware cluster to provision these virtual machines.
  • You will, of course, need a VMware account and a Windows server capable of deploying the Viakoo virtual appliance .ova file to your VMware private cloud.

 

Small Installations (<5000 cameras)

Introduction

Ideally we would want separate storage and compute resources for these installations, but it’s recognized that there likely is insufficient IT resources at these smaller sites to do so in many cases. Where high-IOPS SSD-based enterprise storage is available it is highly recommended that you take advantage of it by adding a 10 gigabit Ethernet card to the compute server and using it to connect to the enterprise storage.  We will, however, describe the stand-alone server scenario since most smaller installations will not have access to enterprise storage.

Stand-alone Server specifications

  • Operating system: ESXi 6.0 or later
  • ESXi-compatible motherboard and chassis. See: https://www.vmware.com/resources/compatibility/search.php
  • Minimum number of processors: 1
  • Minimum number of cores: 16 (may  be implemented as 2x8 core processors)
  • Minimum amount of memory: 64GB
  • RAID controller: Recommended: Broadcom MegaRAID SAS 9380-4I4E Raid controller with CacheVault Flash Module (CVFM04), CacheVault Power Module (CVPM02), or better.
  • Minimum 1U 10-disk (2.5”) storage server chassis with ten 1TB enterprise-class SSD drives formatted as RAID10 (5tb effective)

 

This server should be attached to the video surveillance network with a virtual network preconfigured. The Viakoo virtual appliance .ova file should then be downloaded to a Windows system with access to this server for installation.

Using your existing VMware private cloud

  • Must have access to a minimum of 5tb of enterprise-class SSD storage.
  • Video recorder network must be defined via VMware’s Software Defined Networking and physically connected to your video surveillance recorder network.
  • vSphere account used to provision the virtual machines must have access to the video surveillance network plus must be able to provision the storage needed for the database instance.
  • Five virtual machines are created as part of the installation: GUI (1gb memory, 2 CPU cores), two API (16gb memory, 4 CPU cores),  DBMS (16gb memory, 4 CPU cores), and PuppetMaster (1gb memory, 2 CPU cores). The CPU cores should be 2.4ghz or better. You must have sufficient resources in your VMware cluster to provision these virtual machines.
  • You will, of course, need a VMware account and a Windows server capable of deploying the Viakoo virtual appliance .ova file to your VMware private cloud.

Medium Installations (<20,000 cameras)

Introduction

We do not recommend a stand-alone ESXi appliance with integral storage for these installations because the IOPs requirements are exacting and ESXi’s built-in storage software is built for reliability, not performance. We instead recommend one of these options:

 

  1. ESXi appliance plus a fast flash-based storage server with sufficient storage for both the database and for the virtual machines.
  2. Stand-alone ESXi appliance with sufficient integral storage for virtual machines, PLUS a physical hardware database server with sufficient integral storage for the database.
  3. vSphere cloud plus access to enterprise-class SSD storage.

 

I will document all three solutions.

ESXi appliances plus storage server

This is similar to how we implement our own internal infrastructure to handle this scale.

ESXi appliances:

  • Two recommended
  • HA configuration using VCenter recommended
  • ESXi 6.0 or above.
  • 1U server with 2 rotating disks and Broadcom/LSI hardware RAID 1 on VMware’s Hardware Compatibility List for use installing and booting the ESXi OS (note -- VMware does *not* support “Intel RAID” built into most motherboards).
  • 2 CPU, 3.0ghz or faster recommended
  • Minimum of 24 total cores between the two appliances
  • 96 gb of memory
  • 10 gigabit Ethernet card on VMware HCL, either SFP+ or 10GBaseT
  • Direct-connect SFP+ cable (“DAC”) or CAT6 10GBaseT rated cables to connect the servers to the 10 gigabit switch.

 

Storage server:

  • SSD-based storage.
  • Should be VMware-certified.
  • Dedicated storage OS (commercial, FreeNAS, or equivalent) recommended.
  • May provide NAS shares for VMware data stores, or iSCSI shares.
  • Provides NAS or  iSCSI shares that are used for database data sets. (iSCSI recommended unless VMware-optimized solution like Tintri is being used).
  • A minimum of 12 enterprise-class SSD drives with RAID10 or equivalent.
  • A minimum of 10tb of effective storage space
  • If ZFS based (e.g. FreeNAS), datastore must be created with ashift=13 and 8k blocksize to match PostGreSQL block size.
  • 10 gigabit Ethernet, either SFP+ or 10GBaseT
  • 48gb of RAM for cache and performance

 

10 gigabit switch:

  • Used to connect together the ESXi servers and the storage server
  • Cheapest option is 10GBaseT but current 10GBaseT switches (Netgear) are optimized for use as leaf node switches rather than as core switches thus may not be able to handle a high speed iSCSI / NAS workload.
  • Enterprise-class SFP+ networking equipment is thus highly recommended.

Additional recommendations

 

  • Setting up a storage server requires significant expertise. Unless you have previously set up high-IOPS storage systems for database use (which has different requirements than a read-mostly storage system for general file storage use or a write-mostly storage system for video surveillance storage use), It is highly recommended that you purchase an already-configured storage system meeting the above requirements. Viakoo Inc. is not a storage company and not interested in selling or supporting storage systems. You are best served by buying storage meeting specifications from a company that is interested in selling and supporting storage systems, especially VMware-certified storage systems from companies such as Tintri, TrueNAS, Nimble/HP Enterprise, Hitachi Data Systems (HDS), or Pure Storage. (Note that mention here does not imply an endorsement).
  • The above configuration assumes that if the storage system or virtualization server goes down, some amount of downtime is acceptable while a new virtualization server is commissioned to point at the virtual machines on the storage system, virtual machines are manually moved, or while a new storage server is commissioned with the data drives from the old system. If your installation cannot tolerate that, you will need to improve your redundancy.
  • The storage system can be made redundant via using two storage systems in an HA configuration (note that TrueNAS and Tintri implement this out of the box in an easily configured manner), and dual redundant 10 gigabit switches such as the QCT QuantaMesh T3024-P05. This is easiest if you are using a purpose-built storage system that already has HA built in. Implementing this in the raw Linux OS on a generic server  is possible but non-trivial.
  • Again for best redundancy, you should have two ESXi servers clustered with vCenter with failover capability. Viakoo will automatically recover if virtual machines are interrupted or restarted due to the failure of one node. This also allows transparent vMotion of virtual machines from one server to the other in order to upgrade its OS or hardware.
  • If you have access to an enterprise-class storage system meeting the performance requirements of our solution, it is highly recommended that you use that rather than attempt to create a stand-alone file server that implements the requirements for the Viakoo solution.  

ESXi appliance plus database server

This solution has been tested in our lab supporting over 20,000 cameras but requires significant expertise in order to get sufficient IOPs from the database server. It is recommended only if you have sufficient in-house IT expertise to implement and tune a stand-alone PostgresQL server, but because the PostGreSQL server is running on bare hardware, it is capable of using the full capabilities of the hardware and thus is ideal for high camera count deployments. This also can be cheaper than buying an enterprise data storage system, since the Postgres is running on the same hardware as its data storage thus does not need the ultra low latency of an enterprise grade SAN.

ESXi appliance:

  • ESXi 6.0 or above.
  • 1U server with 2 to 4 rotating disks and Broadcomm/LSI hardware RAID on VMware’s Hardware Compatibility List.
  • RAID array must add up to a minimum of 500gb of storage, preferably 1tb.
  • 2 CPU
  • Minimum of 16 total cores across 1 to 2 physical CPUs
  • 64gb of memory
  • 10 gigabit Ethernet card -- Mellanox Connectx-3 Pro or better
  • Direct-connect SFP+ cable (“DAC”) to connect it to the database server.

Database server:

  • Centos 7,  Viakoo Puppet installation (used to deploy database changes to the database across Viakoo releases)
  • 2U server with 20 2.5” disk slots and SAS3 backplane, or better.
  • Centos 7 compatible SAS3 HBA or RAID controller
  • 10tb or better total flash storage using a minimum of twelve enterprise-class SSD drives in RAID10 or equivalent format
  • 10 gigabit Ethernet card
  • 48gb memory
  • 12 CPU cores

 

Creation and configuration of the database server requires manually installing Viakoo’s custom Puppet client onto a base install of Centos 7 configured with network access, and Internet access in order to download PostgreSQL and related maintenance software from the Internet. Obtaining sufficient performance requires configuring the RAID array to align its write blocks on 8K block alignments, and certain other filesystem creation and possibly special mount options (ashift=13 to align on 8K zone boundaries, setting ZFS to 8K block sizes to align with Postgres block sizes, etc.).

 

For redundancy, two database servers can be configured with the second server being configured as a read replica via Postgres’s built-in log-shipping replication. The second database server will be used as a read replica by Viakoo OnPremise as part of commissioning the cluster.

 

It is recommended that the virtual machines be placed on shared storage, such as on NFS file shares on the database server(s).

VMware Cloud plus Enterprise Storage

You will need sufficient resources on your VMware Cloud to deploy the following virtual machines:

  • Configuration -- 2 cores, 1gb memory, 8gb root volume.
  • GUI - 2 cores, 1gb memory, 8gb root volume.
  • 2x Database -- 8 cores, 16gb memory,  8gb root volume, access via iSCSI or optimized NFS pool to 5tb of enterprise-class storage with minimum of 10,000 iops optimized for database access (10tb and 20,000 iops total).
  • 3x API server --4 cores, 16gb memory, 8gb root volume.

For best performance the iSCSI storage from your enterprise SAN should be presented directly to the database servers.

Large Installations ( > 20,000 cameras)

Viakoo was designed to scale to up to 100,000 cameras in unsharded configurations. It will be necessary to shard the database cluster for installations beyond that point.

The following is assumed:  

  • You have enterprise SSD storage optimized for database use
  • You have a large vSphere cloud capable of large scale deployment of large virtual machines.

The deployment for large installations thus closely tracks the requirements for a VMWare Cloud plus Enterprise Storage deployment for medium-sized installations.

Fundamental requirements:

  • For each additional 15,000 cameras, you will need to add an 16gb RAM additional API instance to process their data.
  • For each additional 30,000 cameras, you will need to add an additional 8tb of storage and  16,000 iops to the database cluster. You will also need to add an additional 8gb of memory to each database instance.
  • 10gbit Ethernet or higher access to the enterprise storage is assumed.
  • Enterprise storage should be provided to the database servers in the form of iSCSI shares on a dedicated storage vSwitch. It is permissible to provide it in the form of multiple iSCSI shares from multiple SAN devices in order to obtain the necessary IOPS, these can then be striped inside the virtual machine to optimize database performance.

 

At some point dedicated database servers become preferable. Being able to cache more of the database in memory via devoting an entire server to the database server speeds up its operations significantly, while not having to deal with VMware’s networking overhead can speed up throughput by as much as 10%. It is still likely preferable that they obtain their storage from an enterprise-class SSD SAN network via 10 gigabit iSCSI or Fiber Channel, however, because enterprise SAN significantly improves upon stand-alone storage both in aggregate performance and reliability.

Enterprise Storage Requirements

Because the Viakoo service requires a high number of IOPs for database purposes, it is important to use enterprise class SSD storage. Saving money here will severely affect the performance of the overall system.

Have more questions? Submit a request

Comments

Powered by Zendesk