Hide Notes 

Slide 1: The Nexus Monitoring System (NMS)™ A White Paper Presented by Nexus Management The Nexus Monitoring System (NMS)™ NOTE: The information contained in this document is the property of Nexus Management and is disclosed to you on the condition that you maintain the information strictly confidential. You are hereby warned that the information disclosed is subject to change without notice or the assumption of any liability on the part of Nexus Management.
Slide 2: The Nexus Monitoring System (NMS) Introduction – The Nexus Monitoring System (NMS).....................................................3 Prerequisites......................................................................................................................4 Product Definition............................................................................................................4 NMS Technology..........................................................................................................4 NMS Technical Product Definition..................................................................................5 Operating Systems........................................................................................................6 Basic Construct ............................................................................................................6 NMS Clients.................................................................................................................7 Nexus Construct............................................................................................................8 BBGDISPLY................................................................................................................9 BBGTRAY...................................................................................................................9 Available TCP/UDP Based Tests.................................................................................9 Alerts and Acknowledgments.....................................................................................11 Reports........................................................................................................................11 Chat Client..................................................................................................................12 Change Control...........................................................................................................12 Information Page........................................................................................................12 Procedures.......................................................................................................................12 System Linked Procedures.........................................................................................12 Adding Hosts..............................................................................................................13 The Nexus Monitoring System (NMS)™ 2 NOTE: The information contained in this document is the property of Nexus Management and is disclosed to you on the condition that you maintain the information strictly confidential. You are hereby warned that the information disclosed is subject to change without notice or the assumption of any liability on the part of Nexus Management.
Slide 3: Introduction – The Nexus Monitoring System (NMS) One of the core competencies of Nexus is the ability to check the status of its client’s networks. This involves; • • • • Server/LAN/WAN uptime Server log checking Disk space utilization Service availability The information is used to; • • • • Generate system health reports Reactively inform people of issues Proactively inform people of emerging issues Predict future IT requirements for clients The development of the Nexus and client Virtual Private Network (VPN) infrastructures has enabled Nexus to access many more clients systems than previously possible, increasing the scope of its network monitoring responsibilities as defined by Service Level Agreements (SLA). The advent of the VPN has also allowed Nexus to consolidate the resources that it has and to take a more systems centric view of managing its client’s technologies, where core product support teams can work remotely on core technologies. So far these have been defined as; • • • • • Networking and VPN’s Microsoft Exchange and messaging NT and W2K Introspect and knowledge management Security With the possibility of; • • • Novell Unix/Linux AS/400 Where as some of the disciplines are vendor specific in term of the software and software platforms deployed many of the hardware platforms differ from client to client and client site to client site. The responsibility of Monitoring and data gathering has traditionally fallen to the operations teams. The operations team has used a number of tools to examine systems; • • • • Connectivity via Ipswich, Whatsup Gold. Daily NT event viewer to look at NT error logs Weekly manual checking of disk space Trend analysis using data from help desk and various manual checks. Availability monitoring has been a 24/5 operations carried out by the help desk in the UK and the US. The Nexus Monitoring System (NMS)™ 3 NOTE: The information contained in this document is the property of Nexus Management and is disclosed to you on the condition that you maintain the information strictly confidential. You are hereby warned that the information disclosed is subject to change without notice or the assumption of any liability on the part of Nexus Management.
Slide 4: To this date there has been no formal method of checking the availability of HTTP, SMTP, DNS, WINS, SMB other than to physically run these tests from a client workstation against the server. There are several issues when looking at the manual processes; • • • • The daily and weekly checks are time consuming, and expensive to both staff and client. The daily and weekly checks although accurate do not accommodate sudden changes in the system environments. The information is also outdated by at least 24 hours. There is no formal process for trend analysis. Many of our competitors, partners and suppliers have the ability to give real-time statistics on various processes taking place on the network. Apart from connectivity, there has been no formal alerting process. The Nexus technical services department undertook a research and development project in April’01 to automate and centralize many of the processes described. Following several months of research, trials and reviews a single product NMS was selected. The product was implemented in the test environment, and then rolled out to the production environment and was kept in a test phase until November’01. During this time the full capabilities of the product have been examined, nxd improvements made to the existing product. The product has been used in new business presentations to new and existing clients and the clients have shown a high degree of interest in the product. Prerequisites Prerequisites for using NMS; 1. A secure VPN Link between the client and one of the Nexus technical centers; • Dornoch, UK :- Europe and the Middle East • Portland, US: - USA and Asia Pacific. * 2. The ability for the clients to browse Dornoch and/or Portland NMS consoles 3. The ability to send TCP packets on port 1984 from the client systems to Dronoch. 4. The ability for Dornoch and Portland systems to have access to all required TCP and UDP ports to perform the tests outlined in Available TCP/UDP Based Tests *Research is underway to find a method of sending 1984 packets over know ports. Product Definition Network monitoring combines the NMS technology, central operations personnel at the GNOC installations and operations procedures. NMS Technology In a nutshell NMS allows Nexus the ability to centrally monitor client networks, the benefits to Nexus are: • The ability to proactively monitor a large volume of systems, without a large number of onsite personnel. The Nexus Monitoring System (NMS)™ 4 NOTE: The information contained in this document is the property of Nexus Management and is disclosed to you on the condition that you maintain the information strictly confidential. You are hereby warned that the information disclosed is subject to change without notice or the assumption of any liability on the part of Nexus Management.
Slide 5: • • • • There is no longer the requirement for routine manual logging and systems checks, such as disk space utilization, CPU utilization and system errors. The ability for system experts to be able to monitors the systems they are responsible for. The ability for client managers to monitor the systems they are responsible for. The ability for people to produce reports on systems they are responsible for. The benefits for the client are; • • • • Reduced cost. The client no longer needs to deploy a large number of Nexus employees to manage their sites. Reduced amount of downtime. Proactive approach to IT budget. By using the trend analysis capabilities of NMS, the client can proactively monitor the use of IT, and to determine key investment areas. A central consolidated view of their global networks. NMS Technical Product Definition NMS is a web based network monitoring portal, presenting a common interface for many well known network monitoring and management tools. These include; • • • TCP/UDP based network tools, such as SMTP, SMB, PING Trend analysis via Multi Router Traffic Grapher (MRTG) Plug into network analysis tools such a sniffers and IDS NMS can be hosted on a Linux, UNIX and NT based systems and has clients available for; • • • • • • • NT/W2K Novell RS/400 VMS/VAX MAC OS Unix/Linux There is a Java version of the client. NMS can be viewed using a web browser, and WAP enabled devices. NMS is open source software and can be customized. This has allowed Nexus to customize the product to current needs of Nexus and its clients. This also gives the product a degree of future proofing where the product can be customized for future requirements. The current system has been enhanced and allows the use of an integrated CHAT system, web based PING, web based Change Request, a web based help system and web based administration of the server. All information gathered by NMS is gathered using port 1984 as defined in RFC1700. NMS has been configured to work on Linux (Redhat 7.2, Professional), NT 4, and Windows 2000 running IIS. The Nexus Monitoring System (NMS)™ 5 NOTE: The information contained in this document is the property of Nexus Management and is disclosed to you on the condition that you maintain the information strictly confidential. You are hereby warned that the information disclosed is subject to change without notice or the assumption of any liability on the part of Nexus Management.
Slide 6: Operating Systems NMS has been configured to work on Linux (Redhat 7.2, Professional), NT 4, and Windows 2000 running IIS. Due to the number of TCP/UDP based tests and packages, stability and performance the Linux version of NMS has been chosen for Nexus Technical Centers. Due to the ease of Administration the NT/Windows 2000 versions of the product is being recommended for client sites, where the client requires a unique perspective of their environment. Basic Construct NMS Display Server (BBDISPLAY) NMS Notification (BBPAGER) NMS Desktop Messenger (BBPAGER) NMS Network Agent (BBNET) Optional For Advanced Monitoring NMS N/T Client NMS Novell Client NMS JAVA NMS Unix/Linux Clients Client NMS Notification MAC, O/S2, AS400 CLients Server Side Components BBDISPLAY, BBPAGER and BBNET are server side components that can be run independently on different servers or on a single server. There can also be multiple instances of BBDISPLAY, BBPAGER and BBNET. These systems can be set-up in fault tolerant and failover configurations. In Nexus the idea is to have a single BBDISPLAY in Portland with multiple BBNET’s in Portland and Dornoch, to allow load balancing. There is also a script for fault tolerance, so when one system fails the other system resumes the function of the failed systems. BBDISPLAY BBDISPLAY is the web based portion of NMS and responsible for formatting the HTML and WAP pages that are viewed. The Nexus Monitoring System (NMS)™ 6 NOTE: The information contained in this document is the property of Nexus Management and is disclosed to you on the condition that you maintain the information strictly confidential. You are hereby warned that the information disclosed is subject to change without notice or the assumption of any liability on the part of Nexus Management.
Slide 7: BBPAGER BBPAGER is the portion of NMS responsible for sending alerts on specific events. Desktop Messenger BBTRAY BBTRAY is a system tray icon that for Microsoft Based systems, which will notify the observer of a change in the NMS status. NMS Clients NT and Windows 2000 version 1.08 A key aspect of NMS is the Microsoft client. The client sends information to the NMS display on; • • • • • • Application, Security and Authentication events as seen in the event viewer CPU Status. Disk Utilization on multiple volumes NT processes Messages from external scripts NT based services. Version 1.08 introduces the ability to start NT services if they stop and to send notifications to the NMS console. UNIX Client The Unix client will work most Linux and UNIX systems including • • • • Verbose Linux messages CPU status Disk Utilization on multiple mounts Kernel processes Novell Client At the present time the Novell client only gives information on Volumes. Additional Clients Clients are also available for AS/400, MAC OS and VAX/VMS. These clients have not been tested at the time of writing this document and will require testing. In addition to the above a Java client is also available, again the use of this client has not been tested. System information can also be gathered using SNMP and MRTG, alarms can then be raised based on MRTG trends. The Nexus Monitoring System (NMS)™ 7 NOTE: The information contained in this document is the property of Nexus Management and is disclosed to you on the condition that you maintain the information strictly confidential. You are hereby warned that the information disclosed is subject to change without notice or the assumption of any liability on the part of Nexus Management.
Slide 8: Nexus Construct Nexus currently has 4 BBDISPLAYS, 3 BBPAGERS, 4 BBNETS. Located in Portland and Dornoch. Client Site D Client Side C Client Side B VPN Internet Connectivity Client Site A Portlan, USA Dornoch, Scotland BBGDISPLAY 10.2.39.13 PORTEST1 10.2.39.62 lmdorlin 192.168.113.30 BBGTRAY 10.2.39.49 EMEA The BBNET (lmdorlin,192.168.113.30) in Dornoch is responsible for data gathering in EMEA. In addition the BBDISPLAY in Dornoch will show all the systems being monitored in EMEA and will also allow staff to run reports and disable systems in EMEA. TCP/IP tests are run every 5 minutes. MRTG Graphs are run every 15 Minutes. USA/APAC The BBNET (portest1,10.2.39.62) in Portland is responsible for data gathering in USA and APAC. In addition the BBDISPLAY in Portland will show all systems being monitored in the USA and APAC and will also allow staff to run reports and disable systems in APAC and USA. TCP/IP tests are run every 5 minutes. MRTG Graphs are run every 15 minutes. The Nexus Monitoring System (NMS)™ 8 NOTE: The information contained in this document is the property of Nexus Management and is disclosed to you on the condition that you maintain the information strictly confidential. You are hereby warned that the information disclosed is subject to change without notice or the assumption of any liability on the part of Nexus Management.
Slide 9: BBGDISPLY This is the global BBDISPLAY (10.2.39.13) unit. There is no BBNET* on this server. All data is passed from EMEA and USA/APAC systems to this server on port 1984, and directly by each NT client. Staff can run global reports. Systems should be disabled on the regional BBNET’s. Host PORTETS1 LMDORLIN Host with BB client BBGDISPLAY in Fault Tolerant Mode. BBGDISPLAY BBNET Network Tests PORT 1984 data flow BBGDISPLAY does not act as a BBNET until there is an outage with LMDORLIN. In this scenario BBGDIPLAY will resume connectivity tests until LMDORLIN returns. TCP/IP tests are run every 5 Minutes. BBGTRAY BBGTRAY (10.2.39.49) runs only connectivity testes. All BBTRAY clients can be pointed to BBGTRAY to test for connectivity. BBGTRAY runs at ever 2 minutes, this test is run against all systems. BBTRAY links to BBGRAY which in turn links to all the other systems. Available TCP/UDP Based Tests NMS has the ability to work with any of the TCP/UDP based tests and Networking programs. The current configurations support the following; • • • • • • Connectivity via PING. CPU Utilization*† Disk Space Utilization* DNS lookups HTTP/HTTPS OS specific System Messages*† The Nexus Monitoring System (NMS)™ 9 NOTE: The information contained in this document is the property of Nexus Management and is disclosed to you on the condition that you maintain the information strictly confidential. You are hereby warned that the information disclosed is subject to change without notice or the assumption of any liability on the part of Nexus Management.
Slide 10: • • • • • • • • • • • • • POP3 and SMTP Telnet. OS Specific Processes*† NT/W2K Services* NNTP LDAP SMB SSH FTP MRTG BBMAP Exchange Monitor VNC * Requires system specific agent. † Not currently available on Novell. The specifics of each test are described PING A single Ping is used to test for connectivity of a service. If the first Ping is unsuccessful 3 further Pings are issued before the system is reported as being unavailable. CPU Thresholds can be set for each system. The Default thresholds are Green < 90 % Amber >90% and RED >95%. A breakdown of responsible processes is given for Microsoft and UNIX type systems. The system will also indicate when a system has been rebooted. Disk Space Utilization Space utilization is displayed per drive. Thresholds can be set for each system. The default threshold are Green <90% Amber >90% RED >95%. DNS Lookup Two modes exist. By default the system uses NSLOOKUP. The system will also use DIG if DIG is activated on the system. HTTP/HTTPS Sites will be tested based on the URL; if the URL is available a green condition will occur. OS specific System Messages NT/W2K: Messages from the NT event viewer are sent to NMS. The client has the ability to filter recursive messages and also to return all events as a green event. UNIX: Messages from the Kernel are passed to NMS. POP3 and SMTP POP3 and SMTP connections are made against the server being tested. Telnet and SSH Telnet connections are made against the server being tested. The Nexus Monitoring System (NMS)™ 10 NOTE: The information contained in this document is the property of Nexus Management and is disclosed to you on the condition that you maintain the information strictly confidential. You are hereby warned that the information disclosed is subject to change without notice or the assumption of any liability on the part of Nexus Management.
Slide 11: OS Specific Processes System specific processes can be monitored. These have to be defined on the OS Specific client. NMS will return a Red condition when a specific process fails. NT and Windows 2000 Services Service availability status is provided. In addition the service can be restarted if the service becomes unavailable. NNTP Test is preformed to show the availability of NNTP services. LDAP LDPA connections are made to check for the availability of LDAP. SMB An SMB connection is made and all available SMB shares are displayed. SSH Testing for the availability of Secure Shell Connections. FTP FTP connection is made to check for the availability of FTP. MRTG MRTG is the acronym for Multi Router Traffic Grapher. This package is used to produce trend analysis for inbound and outbound traffic for any SNMP enabled device. In addition to traffic any attribute of a SNMP enabled system can be monitored and graphed. Fresh holds can be set for each device. VNC To test for the availability of VNC service. Alerts and Acknowledgments NMS has the capability to sends alerts via Pager, SMS and E-mails. However on the basis that VPN technology is being deployed to provide these services, and that VPN’s are subject to latency which can cause false positives, a series of procedures have been designed to help ensure that the correct person(s) are notified of system failure. Acknowledgments can be raised based on e-mails sent our in response to Alerts. Reports NMS has the ability to produce reports. The reports give %; • • Uptime based on connectivity Availability of each service The Nexus Monitoring System (NMS)™ 11 NOTE: The information contained in this document is the property of Nexus Management and is disclosed to you on the condition that you maintain the information strictly confidential. You are hereby warned that the information disclosed is subject to change without notice or the assumption of any liability on the part of Nexus Management.
Slide 12: The reports allow you to look at each event on the system and to analyze these and look at the number of errors contributing to a problem. Chat Client NMS has a built in chat client, allowing secure communication between staff over the VPN. A formal Nexus Policy is needed for the use of this system. The advantages of the nexus chat system lie in its security. All traffic has to travel via the VPN clients and are thus encrypted. Change Control There are two change control forms available on the system. • Additions of a client/service. • Removal of a client/service. At the moment an e-mail is sent to the operations team. In the future an e-mail can also be sent to the Nexus finance sector so that we can bill for the addition of a service and stop billing for the removal of the service. Information Page An additional information page has been built per systems, this included; • • • • • • Locations Function Contact details for administrator Primary, secondary and tertiary contact details for fault reporting. IP Notes on systems Additional links that can be incorporated on this page include; • • • • • Browser VNC Telnet Lights out card APC/TRS power strips Procedures System Linked Procedures A link is provided to the procedures wizard. The procedures wizard is aimed at how operators should respond to the service once a problem is diagnosed. At the moment the wizard caters for; • • • • • • ISP issues Networking and routing issues NT problems Novell problems Linux/Unix problems Exchange problems The Nexus Monitoring System (NMS)™ 12 NOTE: The information contained in this document is the property of Nexus Management and is disclosed to you on the condition that you maintain the information strictly confidential. You are hereby warned that the information disclosed is subject to change without notice or the assumption of any liability on the part of Nexus Management.
Slide 13: Adding Hosts The addition of each host is a 10 minute process, this is carries out during working hours, by operational staff, the appropriate change control form needs to be completed, the addition of each host will be completed within 24 hours, and does not include the weekend. This excludes mass additions, i.e. a new global client. An appropriate project plan would be required. The addition of a host for Analysis via MRTG is currently a 15 – 20 minute process. The Nexus Monitoring System (NMS)™ 13 NOTE: The information contained in this document is the property of Nexus Management and is disclosed to you on the condition that you maintain the information strictly confidential. You are hereby warned that the information disclosed is subject to change without notice or the assumption of any liability on the part of Nexus Management.