AVST Monitoring Guide

Overview

This guide provides information to successfully monitor Applied Voice and Speech Technologies (AVST) CX-E using the Nectar Unified Communications Management Platform (UCMP). This guide assumes a Nectar Remote Intelligent Gateway (RIG) has been deployed on premise and is registered with a Nectar Central Intelligence Platform (CIP) in the cloud. The RIG monitors the health of the AVST CX-E system using ICMP, SNMP, and Windows Remote Web Services Management protocols. One RIG is able to monitor multiple CX-E servers, serving as an aggregation point for multiple alarm sources and processing them through a common event structure.

Audience

This guide is intended for Nectar Subject Matter Experts (SMEs) within a partner who have a strong working knowledge of the partner’s managed service offers, service delivery commitments and SLA obligations. The Nectar SME generally has expertise in infrastructure management and monitoring, and/or Unified Communications with system administration access and a strong working knowledge of Nectar UCMP.

Supported Software Versions

This guide is written for Nectar UCMP client versions 5.5.1 and later.

CX-E Requirements

• SNMP Service
• PowerShell 3.0 or higher with Remote PowerShell enabled (4.0 recommended)
• Service account for Nectar that is a Local Administrator
• Firewall rules to allow ICMP, SNMP, and Remote PowerShell

Nectar RIG Requirements

• Windows 2012 R2 or Windows 2008 R2 (2012 R2 preferred)
• 4 CPU cores
• 4GB RAM (8GB recommended)
• 100GB available disk space (300GB recommended)
• Licensing or patches to allow Remote PowerShell
• PowerShell 3.0 or higher with Remote PowerShell enabled (4.0 recommended)
• CX-E servers added to Trusted Hosts for Remote PowerShell

AVST CX-E System Architecture

A CX-E system is made up of System Servers, Call Servers, and Web Servers. This architecture scales from a single (combo) server with both System and Call Services, up to 3x System Servers, and 20x Call Servers. Web servers are optional and cannot be combined with System Servers or Call Servers.

System and Call Servers are supported on Windows 2008 R2, Windows 2012 R2, and Windows 7. Web servers can be Windows or Linux.

In addition to these servers, there are additional devices that extend the system, such as Dialogic Media Gateways, Perle IOLAN serial-to-IP bridges, and AnywhereUSB bridges,

This chapter contains these sections:

System Servers

Call Servers

Web Servers

System Servers

The System Server is the core of the AVST CX-E system. Every AVST system must have at least one System Server. Clusters of System Servers run in an active/passive state controlled by Neverfail. This third-party software package provides heartbeat, replication, and fail-over.

• Basic system – Combo System and Call Server on one platform
• Dedicated System Server – One System Server on its own platform with no clustering. Call Servers all remote.
• HA System Servers – Two System Servers configured for auto fail-over on the same subnet (for shared IP) and two Call Servers, all at one location.
• DR System Servers – One site with a System Server and Call Server and a second site with another System Server and Call Server. The DR site is on another subnet (no shared IP).
• HA + DR System Servers – Two System Servers configured for auto fail-over on the same subnet (for shared IP) and a third System Server at a remote site (no shared IP).

The system server is made up of a core group of Windows services, plus optional services that are enabled with extra licensing.

In a clustered environment, CX Background Service, Neverfail SCOPE Data Collector, and Neverfail Server R2 services are started on all active and passive System Servers. The remaining services are started only on active System Servers, and stopped on passive System Servers.

System Server Core Services

System server core services include:

System Server Optional Services

System server optional services include:

Call Servers

Call Servers provide the telephony resources and PBX integration of the AVST CX-E system. Every AVST system must have at least one Call Server, and could have up to 20. This could be collocated with the System Server on one platform when the System Server is not clustered.

Usually Call Servers are installed on dedicated servers and located close to PBX resources for integration.

Call Server Core Services

Call server core services include:

Call Server Optional Services

Call server optional services include:

Web Servers

AVST web applications typically run on customer provided web servers where the OS and machine are managed by the customer. These can be physical or virtual, Windows with IIS or Linux with Apache, internal network or DMZ, single server or clustered. Ideally Nectar would monitor the website to confirm its online, and leave lower level monitoring of services and OS to the customer.

Web Phone Manager

This website provides a web front end to subscriber mailboxes, allowing access to settings and messages. It’s usually installed on a web server on the internal network but sometimes also put in the DMZ. The website is configured with a default document of index.php which prompts the user with a login screen. An HTTP/HTTPS call to the URL should return the login page.

Example: https://wpm.avst.com

CX Mobile

This website is a service end point for the AVST CX Mobile application, which runs on Android and Apple phones. It’s usually installed on a web server in a DMZ and has a publicly accessible URL. Users do not access the site with a web browser, but it can be browsed to verify the site is online. An HTTP/HTTPS call to

Example: https://cxm.avst.com/cxmns.php

Mobile Admin

This website provides a web front end for basic administration. It’s usually installed on the same web server as Web Phone Manager, as a virtual directory, but could be configured as a separate website. The most common URL would be https://. An HTTP/HTTPS call to the URL should return a login page. There is not externally accessible example for this since it is rarely put in a DMZ.

Configure AVST CX-E Monitoring with Nectar

Configuring AVST CX-E monitoring with Nectar involves performing these steps:

Import the Poller Pack

Enable SNMP on Each CX-E Server to Be Monitored

Create Agent for Each CX-E Server to Be Monitored

Run Host Wizard to Add Services to Be Monitored

Configure CX-E Windows Event Log Polling

Import the Poller Pack

Before configuring the CX-E Servers in UCMP, you must install the AVST poller pack, which you will obtain from Nectar. This AVST poller pack will contain the function called ms_svc_state, referenced in Run Host Wizard to Add Services to Be Monitored.

When you have the AVST poller pack, follow this procedure:

1. Access your RIG.
2. Navigate to Configure > Poll Functions.
3. Right-click in the Poll Functions pane and select Load Pack. The Open dialog window appears.

Figure 3-1Load AVST Poller Paclk

1. Locate and select the AVST Poller Pack from the location where it was saved (i.e., Documents folder, Desktop, etc.)
2. Click Open.

For more information on importing poller packs, see the “Manage Poll Functions” chapter of the

RIG Administration Guide.

Enable SNMP on Each CX-E Server to Be Monitored

These are high-level instructions. For more details on this procedure, see Microsoft documentation on using the SNMP Service Properties window.

1. Disable the local firewall.
2. Install the SNMP Service if you have not already installed it.
3. Right-click on the installed service and select Properties. The SNMP Service Properties window appears ( Figure 3-2).
4. Select the Security tab. Configure the SNMP Service by creating a community with read-only privileges and adding the RIG as a trusted server.
5. Select the Traps tab. Configure Traps to be sent to the RIG.
6. Click Apply.

Figure 3-2SNMP Service Properties > Security & Traps Tabs

Create Agent for Each CX-E Server to Be Monitored

Follow these procedures to create each required CX-E server that you want to monitor in Nectar:

1. Log in to the CIP.
2. In the Location panel on the left, select the RIG that will monitor the CX-E server.
3. Navigate to the menu bar and Select Health > Elements.
4. In the Elements window, select Agents, right-click on the list of agents, then select Add.

Figure 3-3Add Agent

In the Add Agent dialog window ( Figure 3-4), enter the following credentials:

• 
  
  1. In the Name field, enter a meaningful name for the CX-E server (e.g. System Server).
  2. In the IP field, enter the IP address of the CX-E server.

Note

Enable polling of CX services only on the agent using the Public IP. For the agents using the management IPs, select CX Background Service and Neverfail Server R2.

• 
  
  1. Select the appropriate SNMP Version radio button — dependent on your security policies.
  2. In the Community field, enter the SNMP community name as configured on the CX-E server.
  3. Click OK.

Figure 3-4Add Agent

1. Repeat these steps to add an agent for each additional CX-E server to monitor.

Note

If monitoring multiple system servers, create an agent for each individual server using the Management IP address. Then, create another agent using the Public IP address.

1. To configure the RIG to monitor Windows events, see Configure the RIG to Monitor the Windows Event Log for CX-E Events.

Run Host Wizard to Add Services to Be Monitored

For this section, refer to System Servers and Call Servers for the list of services to be added for each agent you created in Create Agent for Each CX-E Server to Be Monitored.

1. Log in to the CIP.
2. In the Location panel on the left, select the RIG that will monitor the CX-E server.
3. Navigate to the menu bar select Health > Elements.
4. In the Elements window, select Agents, then right-click on one of the CX-E server agents that you created earlier in the All Agents pane.
5. Select Host Wizard from the drop-down menu.

Figure 3-5Host Wizard

1. In the Host Wizard > Create pollers for Agent resources dialog window ( Figure 3-6), make the following selections:
   • Create pollers to monitor CPU status - Select this check box to create pollers with this host wizard that monitor your CPU status.
   • Compression - Select a compression (data retention time-frame). Your choices are 1 Week, 2 Weeks, 1 Month, or 3 Months.
   • Host Operating System - Select your host OS. Your choices are Linux, OS X, Windows, or

Solaris.

Note

The operating system is automatically detected by the RIG according to the file system formatting and process/service list returned from the host.

• 
  
  • Click Next.

Figure 3-6Host Wizard - Create Pollers for Agent Resources

1. In the Host Wizard > Disks/Memory window ( Figure 3-7), select the disks/memory for which you will create pollers - use the CTRL key to select multiple items.
2. Click Next.

Figure 3-7Host Wizard - Disks/Memory

Note

When Windows is selected for the Host Operating System option in the Host Wizard - Create pollers for Agent resources page, the Host Wizard - Services page appears.

When any other operating system is selected, the Host Wizard - Processes page appears instead, where you can select processes instead of services.

1. In the Host Wizard - Local RIG > Services window:

• For Windows, select all services beginning with “CX”. Use the CTRL key to select multiple services.
• For Linux/Solaris, select all services beginning with “AT_”. Use the CTRL key to select multiple services.

Note

The list of services/processes that appear vary depending on the agent that you are working on.

1. Click Next.

Figure 3-8Host Wizard - Services

1. In the Host Wizard - Layer 3 Status (Ping) window ( Figure 3-9), select the IP addresses for which you want to obtain statuses - use the CTRL key select multiple addresses.
2. Click Next.

Figure 3-9Host Wizard - Layer 3 Status (Ping)

1. In the Host Wizard - Layer 2 Status window ( Figure 3-10), select the appropriate interfaces - use the CTRL key select multiple interfaces.
2. Click Finish.

Pollers are now created according to your selections. Repeat the process to create a second agent.

Figure 3-10Host Wizard - Layer 2 Status

Configure CX-E Windows Event Log Polling

Windows Event Log polling requires Remote PowerShell access from the Nectar RIG to the CX-E server. If using Windows 2008 R2, PowerShell must be upgraded to version 3.0 or later. Version 4.0 is recommended.

Install and Configure Remote PowerShell on Each CX-E Server to Be Monitored

1. On Windows 2008 R2 based servers to be monitored, upgrade to PowerShell 4.0:

https://www.microsoft.com/en-us/download/details.aspx?id=40855

1. Enable Remote PowerShell:
   1. Open an Administrator Command Prompt.
   2. Run winrm quickconfig.
   3. Follow prompts to enable remote management and create firewall exceptions:

https://technet.microsoft.com/en-us/magazine/ff700227.aspx

https://msdn.microsoft.com/en-us/library/windows/desktop/ aa384372%28v=vs.85%29.aspx

1. Create a Nectar service account:
   1. This can be a domain account or local account.
   2. Add the account to the local Administrators group on every CX-E server to be monitored.

Configure and Test Remote PowerShell Access from RIG

1. On the Nectar RIG, add the CX-E server to be monitored to the Trusted Hosts list.
   1. Open PowerShell as Administrator.
   2. Add the CX-E server IP address to the Trusted Hosts list by running these commands:

Set-Item WSMAN:\Localhost\Client\TrustedHosts -Value

-Concatenate

Get-Item WSMAN:\Localhost\Client\TrustedHosts

1. Start a Remote PowerShell session to validate connectivity by running this command:

Enter-PSSession -ComputerName

1. Enter the password of the
2. Retrieve event log entries for AVST processes by running this command:

Get-EventLog -Logname Application -Source AT_* -Newest 10

Configure the RIG to Monitor the Windows Event Log for CX-E Events

1. Log in to the CIP.
2. In the Location panel, select the RIG that will monitor the CX-E server.
3. In the menu bar select Health > Elements.
4. In the Elements window ( Figure 3-11), select Agents..

Note

Configure Windows Event Log monitoring only for the agents using Management IPs. DO NOT configure Windows Event Log monitoring for the agent using the Public IP.

1. Select and highlight the appropriate CX-E server, then right-click and select Edit.

Figure 3-11Edit Agent

1. In the Edit Agent window ( Figure 3-12), select the Properties tab.
2. Click the Add button to add the following Remote PowerShell login information. You must repeat this process four times to add all four lines:

Note

Your AVST engineer should provide you with the PowerShell information required here.

• 
  
  • powershell.domain = domain
  • powershell.host = IP address of CX-E server
  • powershell.user = username
  • powershell.password = password

1. Click OK in both windows to accept the changes.

Figure 3-12Add PowerShell Login Information

Note

If the Nectar service account is a local account, set the powershell.domain value to the IP address of the CX-E server.

1. Navigate to the menu bar and select Configure > Windows Events.
2. In the Windows Events > Agents pane, click Add.

Figure 3-13Windows Events > Agents

The Add Agent dialog window appears ( Figure 3-14):

Note

The RIG will only display agents that have working remote PowerShell capabilities. If the CX-E server is not shown in the list of agents, review the previous steps to install, configure, and validate remote PowerShell functionality.

1. Select Application from the Name drop-down list.
2. Select Error and Warning from the Event Type list.
3. Select Last 1 Day from the Event Time Range list.
4. In the Event Source Filter field to enter AT_*.
5. Leave Event Message Filter field blank.
6. Click OK.

Figure 3-14Windows Event - Add Agent

Configure AVST CX-E Alerting with Nectar

The default Dependency Trees for each CX-E server Agent show PING and Windows Service (SNMP) polling results. Any failure will display an Alert level of Critical.

Configuring AVST CX-E alerting with Nectar involves performing these steps:

Configure Aggregate Alarming for CX-E System

Configure Aggregate Alarming for CX-E System

To configure an aggregate alarming for the CX-E system:

1. Log in to the CIP.
2. From the Location panel on the left, select the RIG that will monitor the CX-E server.
3. In the menu bar select Configure > Events > Aggregate Alarms.
4. In the Aggregate Alarms > Alarms pane, select and highlight the appropriate Aggregate Device Name; then, click Add.

Figure 4-1Aggregate Alarms

1. In the Add Aggregate Alarm dialog window ( Figure 4-2), enter or select the following credentials:
   1. Name: enter a name for the aggregate alarm (e.g. AVST Applications).
   2. Function: select max.
   3. Event Type: select global.

Figure 4-2Add Aggregate Alarm

1. Double-click the green IF circle to display the Add Condition dialog window ( Figure 4-3):
2. Enter the following values as shown in the figure below:
   1. Variable: select displayName
   2. Function: select contains
   3. Value: enter AT_
3. Click OK.

Figure 4-3Add Condition Window

1. You can now view the AT_* alarms by navigating to Health > Events and viewing the Events

panel:

Figure 4-4View Alarms