How to develop monitoring solutions for large networks

Have you ever wanted to experience what it’s like to be a network administrator in a multinational corporation? Or, on the contrary, do you already know what it’s like to be a network administrator, but frequently find yourself having to test different deployment scenarios before putting them into the live network environment in order to avoid unnecessary problems? In both cases, we have good news for you – Virtualization and increasingly accessible cloud solutions make virtually any network simulating possible in just a few clicks.

We interviewed Jiří Pokorný, technical lead developer at SolarWinds, about the potential of virtualization and its vital role in the development of enterprise-grade monitoring software, and its necessity of testing in a production environment.

SolarWinds is currently the world’s market leader in network monitoring systems for network administrators. Thousands of IT administrators around the world use its solutions. Let’s take a sneak peak of the company’s product development processes.

Are you solely using virtualization to test new updates during the development, or is this method used for something else?

We have built our virtual lab for testing various scenarios for networks ranging from tens to thousands of servers. That represents our customers’ infrastructure. It’s not just about testing new updates of our product releases. Sometimes we need to simulate the environment of a particular customer who has a problem or needs technical advice. With virtualization, we can simulate their network as it is, analyze it, test their scenario to give them advice on how to optimize their particular IT infrastructure to achieve better performance without further investment in new hardware or software.

How do you simulate complex networks?

There are many tools and ways to do that. When it comes to testing specific programs running on various operating systems, virtualization tools like VMware® vSphereor Microsoft® Hyper-V® can be used. To test specific applications, Docker® is the great choice not only for its open source distribution. For network infrastructure simulation including virtual routers, switches, and other network elements, we use the GNS3 emulator. SolarWinds bought GNS3®in 2015, but we’re offering it for free under an open source license.

So there is no universal or a single one solution for your work?

Computer networks have become so complex that it is hard to imagine any magic and universal solution that can simulate every possible scenario out there. It always depends just on what exactly do you want to test. It can be generalized and categorized into three basic schemes: a one-time simulation of a network, a general network setup for software development, and a test environment for common scenarios replicating a large number of different network types and set-ups for performance testing.

Would it not be easier to have a few servers, routers, workstations with Windows® and Linux®?

It looks simple and logical at first glance – three operating systems, three machines being tested in the lab connected to one network. However, each OS has several supported versions. Just consider many different versions of Linux distributions that differ from one to another on many levels. Additionally, the particular program may run on a system influenced by specific configuration and OS version, or communication between consoles running on different types of operating system – all this has to be included in your test environment scenarios. Now we are talking about an environment containing dozens of physical servers and workstations. In this case, it is easier and, of course, more economical to virtualize. This solution allows us to test complex system of network elements regardless their physical availability. For larger projects, even virtualization is unfortunately quite hardware-intensive because each virtual machine allocates a portion of the memory and several GB of disk space. But compared to building, maintaining and upgrading a “real” testing lab, it is much more cost effective. Virtualization has some disadvantages as well; it is not possible to test anything that depends purely on hardware health such as temperature on the processor.

Is there a way that is not so much demanding for system resources?

If you do not need to test operating systems, but rather individual running applications, you are able to choose just an application virtualization. To do this, the Docker program, for example, distributed as an open source, supports a wide range of operating systems (despite the fact that MS Windows is on the list of supported applications for a relatively short time). Every application within this virtual environment is running as if it running on a stand-alone system so that individual applications do not interact with each other by rivaling the same system resources. The main advantage is that you are able to simulate this environment very quickly. Usually, you do not have to install or configure anything, just download the pre-installed package from the Docker portal.

What can be tested in an environment like that?

Perhaps every network administrator wants to test a new app before putting it into a live operation to make sure there are no conflicts. But it needs a test lab/environment, ideally separated from the corporate network. If it does not, it poses a risk of application malfunctioning, installation corruptions, or conflicting with other applications within the network. Making your backups before installing the application and then re-rolling them back as soon as any problem occurs is not an elegant solution. Instead, testing the program in virtual space is fast and safe.

Does this solution have any limitations?

Yes, unfortunately, the application exists in a vacuum. This means that do not take into account the network topology or some hardware elements like employed disk arrays. However, this is one of the ways in which testing should not be forgotten. Some problems with the applications and its settings can be solved much faster and with minimal cost.

Your business is focused on developing tools for monitoring networks, servers, and running applications within those environments. Different infrastructure set-ups had to take into account during tests. What do you use in this case?

It is true. Of course, we use the previous variants for various reasons and scenarios, because as I said, there is no universal solution, but the most common testing tool we use to emulate large network infrastructure is the GNS3 emulator. This tool allows us to simulate the entire network and build it from scratch. Probably everybody who studied computer science remembers network lessons where you were learned how the network protocol works. You had to install several operating systems, configure the network adapters, and then start experimenting. In the case of expensive hardware, like Cisco routers, or complex network setups, you were not able to reach them without actual hardware. All this can be simulated today in the emulator.

How does it work?

You can add any number of various virtual devices to your simulated network via a plugin system. Individual devices can then be connected using a “cable” simply by linking two network elements in a diagram with a drawn line. The connection simulates the use of telnet or SSH® protocols, and each element behaves like a real piece of hardware. This allows you to test application’s behavior in a dynamically changing network infrastructure, simulate metrics such as lost packets or high latencies, and much more. For our testers, it is an indispensable tool.

How big was the largest network you have ever simulated?

The biggest network we have tried to reproduce was consisted by thousand servers and other network elements. To give you an example, the GNS3 is used by companies such as  Walmart® or AT&T® to simulate their complete IT infrastructure. As far as I know, NASA uses this environment to test their updates before actual deployment to their network as well.

What would you like to say to those who would like to try network virtualization?

Go for it! Simulating complex network behavior has never been easier. Additionally, there are a number of examples of ready-made topologies available online, which can be simply downloaded. In addition, just imagine the hiring process for an administrator/tester/developer position, there is always a question about your experience with those environments. And how many people can say that they know how the topology of a multinational network consisting of thousands of servers looks like? At SolarWinds, we cannot imagine our professional life anymore without GNS3.


Biography: Jiří Pokorný

Jiří works as a programmer and teacher of .NET C#. He focuses on QA and continuous delivery. He teaches C# language at VUT and Masaryk University in Brno. He’s an active member of Windows User Group and has held many lectures in the Czech Republic.

SolarWinds Czech

SolarWinds Czech, based in Brno, is the main development location of technical software for network administrators and system administrators. According to IDC’s independent survey, the company, at the end of last year, has become the world leader in network monitoring. The company’s products have received many awards from critics and industry professionals.