Comparison and evaluation framework

Comparing and evaluating cybersecurity solutions for business

This solution evaluation template provides a comprehensive set of aspects that should be enough in most vendor selection scenarios. Still, if additional requirements are needed, it is easy to add other evaluation attributes to existing sections.

This framework can be used as a guide, i.e., a list of questions that should guide the conversations and demo sessions with solution providers. It also helps to maintain objectivity in the review process as the same principles are applied equally to all potential vendors.

The evaluation framework has eight sections that act as a guide for analyzing potential solutions.

Let's take a closer look at each of the sections.

Background info and track-record

This section acts as a background information depository if you need to share your evaluations and comparisons with your team or if a few products are bordering on their final score. It might come in handy if you need to rely on business-level information to make the final decision.

Product assessment

Technical quality is traditionally approached from 1) service availability, 2) scalability, and 3) configuration and maintenance perspectives. It gives a general assessment of likely future company resource investments in using and maintaining the solution and potential additional costs due to changes in the scale of use or possible downtimes.

Not all solutions available in the market are comprised of a single product. You will likely be comparing monolithic alternatives to solutions that integrate proprietary and third-party products to deliver the same functionality and outcome. You might consider assessing product integrity by examining the following categories: identity, device and network management.

Organization-fit

You can choose the most potent tool and lose more than gain. The idea of the organization fit is a match-up between the complexity and requirements of the solution and users' organization capacity. Excess functionality can easily create barriers to everyday use and require additional investment over time. On the other end of the spectrum, simplistic solutions can fail to satisfy the basic organizational needs or quickly show growth pains.

We invite you to consider whether the tool or solution satisfies your actual use-case(s) in this area. It is critical to investigate what resources and competencies the solution will require and how well it matches the organization's cybersecurity maturity level. From a technical perspective, setup and integration costs should also be assessed.

Functionality overview

We suggest evaluating features in 4 essential product component groups: cloud gateways, endpoints, network filtering, and administration. Each of these groups has relevant features that can be reduced or expanded in the way that you see fit.

Function related benefits

This part of the tool is dedicated to marking and evaluating specific outcomes and added value that the vendor or product under question can provide. This information is usually covered in vendors' sales materials or can be directly discussed during live meetings.

Credibility

The section on credibility covers the standards and compliance aspects of the service. This criterion is likely to be important for your legal and compliance teams. Let’s suppose your company does not have specific compliance requirements. In that case, this section should still be evaluated as the availability of standards and adherence to regulatory frameworks means that the vendor is committed to long-term perspective and business continuity.

Support and services

Account management services, as well as support and product documentation quality, might seem like secondary elements. Still, they indicate a long-term vendor commitment to provide a fully developed service that accounts for IT team needs.

Pricing

The pricing approach should not be left out of the early vendor evaluations as it can become a significant headache after choosing and adopting a solution. It is crucial to assess the general pricing model attractiveness and clarify additional pricing policies, especially in cases where the core product and optional additions lack transparency. Contractual lock-in is likely to be an important topic when discussing solutions with management, so pay extra attention to pricing flexibility.

What’s next?

There are two fundamental ways to deal with the final scoring. The primary option is to calculate all the points and declare a clear winner. The second option – change weight ratios for the aspects you feel are more significant for your organization. You can evaluate which vendor delivers a better match and compare it with their price offering.

Even if you are short on time to do a total scoring of all the vendors and solutions you are reviewing, you can still use the same template. It might serve you as a reminder to ask critical questions and make comments and notes for later review and decision-making sessions.