Reducing Service Outages using Observability for Hybrid-Cloud Environments

Agentless Hybrid-Cloud Observability using the cCloud™ Visibility Suite

Network metrics and streamed data from network monitoring increasingly provide valuable insights into network flow behavior and anomalies. The deluge of data can be overwhelming, and the shared responsibility for security in the cloud access to network telemetry remains critical to compliance and operational efficiency. The operations team can reduce service outages by adding instrumentation via Network Virtual Appliances (NVA) in the cloud environments to provide agentless packet capture and offload replication services from the production workloads. Network Virtual Appliances coupled with other cloud native services are available to provide insight from network packets. These appliances provide visibility services by replicating and delivering packets to security tools, network performance tools, and dashboards that are key to reducing service outages.

The cCloud™ Visibility Suite provides visibility and observability ready-to-deploy support for Amazon Web Services, Microsoft Azure, and Google Cloud public cloud infrastructure. The suite is an integrated set of components that perform packet acquisition, replication, forwarding, packet capture-to-storage, and analytics. Altogether the suite provides vital monitoring, visibility, and observability for cloud infrastructure without the management overhead and additional security risks of placing agents or probes into the production workload host, virtual machine, or application layer.

The cCloud Visibility Suite consists of these components:

cClear®-V Analytics Engine that provides network health and traffic analytics, visualizations, and alerts

cStor®-V Virtualized Packet Capture-to-Storage and analyzing network traffic

cVu®-V Virtualized Network Packet Broker (NPB) including packet acquisition, replication, forwarding, and delivery to analytics, tools, and dashboards

Figure 1 below shows an example of a simple environment for subnet monitoring using the virtual appliances. Two subnets, “Prod” and “Default” simulate production East-West traffic with a separate subnet “Monitoring” for the tool’s infrastructure. Microsoft Azure was used in this example, however the methods apply to essentially any public cloud infrastructure.

Figure 1 – Cloud Subnet Monitoring Lab Setup Example

The traffic is routed via User Defined Routes using the Azure Route Table between the subnets. The virtualized network packet brokers provide replication and forwarding services to all packets routed to the Azure Load Balancer. The Virtualized NPB replicates capture appliance routes packets to storage for historic forensics, replay, and exporting as streams and PCAP files.

Use Case #1 – Isolating Subnet Connectivity (East-West) Issues

Let’s simulate excessive traffic to the Device Under Test (DUT) view the workflow and outcome through the cClear-V Analytics Engine web interface.

Use Case: Isolating Subnet Connectivity (East-West) Problems
Description: An unknown issue is reported in lab-east2-vnet/default
Simulation: – Source Host 10.3.0.4 in subnet lab-east2-vnet/prod
– Traffic Type Flooding HTTP on port 8080
– Destination Host 10.3.1.4 in subnet lab-east2-vnet/default
IT Operational Response: Need to see network health information, including visualizations of TCP Analytics for the subnet traffic over the last 30-minutes
Workflow: Directly login to the cClear®-V virtual appliance and choose the Dashboard TCP Health Level

Figure 2 shows that the “New Sessions” KPI has 58268 open sessions. The TCP Health Level 1 dashboard shows each row representing each network segment. Prod and Default subnets show the “New Sessions” KPI and are triggered red, with the Monitoring segment displaying minor counters (orange).

Figure 2 – TCP Health Level 1 Dashboard

Click on the red default New Sessions box for further details.

Figure 3 below shows the Syn Metrics Network Monitor Level 2 dashboard with details for the “New Sessions by Server” highlighted in red, the destination host IP 10.3.0.4, and the port number 8080 indicating the New Sessions Application.

Figure 3 – SYN Metrics Network Monitor Level 2 Dashboard

Outcome: In this use case, the operator used dashboards presented by the cClear®-V Analytics Engine to gain insights with just a few clicks to isolate the network segment with the issue, the IP addresses involved, and the destination port of the traffic.

Accessing Stored Packet Capture for Network Forensic Analysis

In production environments, numerous cStor®-V Packet Capture-to-Storage appliances are deployed in multiple locations strategically placed in the network topology. Hence, accessing each cStor-V web interface is not practicable. The cClear®-V enables downloads across multiple cStor-V appliances for a merged and filtered forensic view downloading the PCAP file.

Use Case #2: – Capturing Subnet Traffic for detailed Forensic Analysis
Description: An issue is reported in lab-east2-vnet/prod including approximate time (Refer to Figure 1)
Simulation: – Source Host 10.3.0.4 in subnet lab-east2-vnet/prod
– Traffic Type Flooding HTTP on port 8080
IT Operational Response: The subnet packets for the last 2 minutes from 12:30 pm are needed
Workflow: Directly login to the cClear-V virtual appliance, group packets for the timeframe required, export as a PCAP file, and analyze using Wireshark

Select “Capture” and choose your options:

Range Settings: Start/End
From: 2022-01-26 12:30:00
To: 2022-01-26 12:32:00
Maximum Download Size: 10MB (first)
Filter Type: Fast (all packets)

Click on “Start Download” to transfer the PCAP file to your local computer. After the transfer is
completed, open it into Wireshark for analysis.

Figure 5 – Analysis of Captured Packets

Figure-6 shows Wireshark displaying the traffic with the source address 10.3.1.4 generating HTTP traffic to port 8080 on destination 10.3.0.4.

Figure 6 – Wireshark PCAP Forensic file

Outcome: In this use case, the operator selected, grouped, and exported a specific set of captured
packets as a PCAP file for analysis using Wireshark. The analysis shows the source 10.3.1.4
generating HTTP traffic on port 8080 to the destination IP 10.3.0.4

Summary of Reducing Service Outages with Observability for Hybrid-Cloud Environments

At cPacket Networks, we understand observability for hybrid-cloud environments (as well as for on-premises, single-cloud, and multi-cloud environments) and how monitoring subnet traffic the cCloudVisibility Suite helps you reduce service outages and accelerate incident response. In this real-world example, we were able to solve the root cause using the agentless architecture and the use of Network Virtual Appliances with visualizations and dashboards. Good Hunting!

Further use cases and details can be found in the Network Visibility Troubleshooting – Application Note here: