Our partner organization is a mid-size cybersecurity organization headquartered in the US. The firm specializes in network traffic analysis and its products help clients to have a secure experience over their networks by detecting and preventing malicious activity & security threats. It taps into the power of Machine Learning and Threat Intelligence to analyze millions of exchanges on networks and is amongst the top 10 for its robust security solutions.

In today’s world where everything is happening digitally, an unreliable network can literally bring a business to a standstill. For maintaining a healthy data center, network monitoring solutions are important so that companies can have better management and control over their networks.

These solutions can give companies the flexibility to keep a track of and analyze their networks for troubleshooting & detecting any mistrustful activities in real-time.

Our partner organization wanted to enhance its network monitoring solutions with a protocol surveillance tool. System administrators were trying to find a solution so that they didn't have to manually extract and analyze activity log files generated by these protocols on the server.

1. Monitoring the running activities of the protocols

• The Active Directory does not make available the file location of the logs in the local file system which poses a technical challenge to access these logs.

• When Apache Kafka was used on the Windows server, it was found to be incompatible with the Rust programming language and was unable to produce logs on the message queue.

• There were unavoidable roadblocks while compression of logs before they could be produced on Kafka topic

The solution came in the form of Prolance, a protocol surveillance tool that Knoldus built on the Rust programming language. Let’s understand what this project enabled:

1. Monitoring the running activities of the protocols

Prolance was build to monitor the activities of the following network protocols -

• Dynamic Host Configuration Protocol (DHCP) - This is a network management protocol that can dynamically assign IP addresses to client devices.

• Active Directory (AD) - This is a Microsoft technology that allows network administrators to generate & manage domains, users, and objects along with their permissions and access control within a network.

Prolance automated the extraction and analysis of the DHCP audit logs which monitor the network devices and prove to be an invaluable diagnostic tool when security is compromised over the network. Similarly, AD logs provide an insight into possible abuse of access rights & privileges.

2. Filtering out the logs as per monitoring requirements

Once we started processing these logs, we had two scenarios - either the user requires raw logs of the protocol or logs can be filtered as per the monitoring requirements. Take, for instance, the scenario where the user is interested in obtaining only the IP addresses and the device name from thousands of DHCP logs. Filtration will be required so that only the relevant pieces of information (like Discovery, Offer Request & Acknowledge phases) are extracted as per user requirements.

3. Continuous streaming of the logs onto the Kafka Topic after compression

Whether the logs are filtered or used raw, they go through the process of compression before being produced onto the Kafka topic. Apache Kafka has been used here for message queuing which lets you scale your processing. This happens as Kafka enables you to divide the processing over multiple consumer instances. We used the gzip compression technique and enabled the user to schedule this process as per requirement.

• Two-phase filtration technique along with parsing

To solve the first challenge pertaining to AD logs, we employed a two-phase filtration technique. The first phase involved the extraction of the AD logs from the security files present on the Windows server and the second phase was filtering out the relevant ones from the AD logs as per user requirement. Since the security files existed in the .evtx format, we had to use an evtx parser to change the format into a usable one.

• Ensured compatibility of Kafka & Rust

To solve the problem of incompatibility between Apache Kafka and Rust, we disabled certain default features of Kafka which were conflicting with the Windows server.

• Enabled compression of logs

After we disabled the default features of Apache Kafka in the previous step, it posed problems in compression of the logs. To remove this roadblock, we had to enable one of the default features of Kafka pertaining to compression.

1. Enhanced network visibility as system administrators could now track and analyze network activity of the devices present on the network.

2. Timely prevention of potential harm to the networkas Prolance allowed keeping a check on malicious logins on workstations or illegitimate software installations/data transfers.

3. Significant reduction in the Mean Time to Detect (MTTD) and Mean Time to Resolve (MTTR) as security teams were able to respond faster to security breaches