In this post rather than cover Kaspersky chicanery I instead focus on a pretty big problem the Security industry at large faces. This is a classic example of a well-intentioned idea gone horribly wrong.
I was reminded to post this because of a recent posting by a security company called Carbon Black. In its posting at http://www.carbonblack.com/second-av-study-reveals-small-window-for-catching-new-malware/. I quote
"We downloaded 84 random malware samples from malc0de.com and submitted each to VirusTotal to see what each antivirus product had to say about them."
I was reminded to post this because of a recent posting by a security company called Carbon Black. In its posting at http://www.carbonblack.com/second-av-study-reveals-small-window-for-catching-new-malware/. I quote
"We downloaded 84 random malware samples from malc0de.com and submitted each to VirusTotal to see what each antivirus product had to say about them."
I see this all the time - you have some upstart company like Carbon Black (who I had never heard of), wanting to promote some cool new technology of their and they do this by undermining "traditional Anti-virus" by quoting some bogus results from VirusTotal. If I got a dollar for every time I heard the term "Traditional Antivirus can't/doesn't" at a conference, or in a blog or security paper, I would be a very rich person. Guys, read my lips "THERE IS NO SUCH THING AS A TRADITIONAL AV PRODUCT ANYMORE".
I can understand where Carbon Black is coming from - With all hacks, APTs, infiltrations, nation-sponsored cyberattacks in the news, there is a lot of money to be made being a security professional. And let me tell you, self-professed security pros/companies are crawling out of the wood-work. The result, security pros are a dime a dozen and everyone is trying their best to stand-out, be heard, gain some attention, fame, notoriety, call it what you may. These security pros/companies do everything they can to sensationalize even the most mundane issues. In the process they often demonstrate how their poor understanding of the basics of today's security challenges and products. And worse of all, it doesn't seem matter who they are. I have seen this problem from some iconic WallStreet Journal tech reviewers to your friendly neighborhood PC magazine editor, all appear to have an outdated impression of today's security products and assume they work exactly like they did 10 years ago.
Case in point - Quoting VirusTotal.
For the less astute reader, VirusTotal is a free website (www.virustotal.com) that allows you to submit a file and the website will scan that file with security products from 45 security vendors and return the results to you in a neatly formatted table that allows you to compare the results of competing vendors. A good idea of considerable value to the community but as you shall see it has little value for any security pro that really knows their stuff. Sometimes I wish it never existed. Due to the simplicity of this website and the very fact that its free, it has become the go-to site for a lot of security pros including IT administrators to compare the effectiveness of the current security product they own versus another one that they have heard "good things about". Not surprisingly they will sometimes use virustotal results as "evidence" to go beat up their security vendor (say McAfee), telling McAfee "Hey McAfee you are absolutely useless. Look how many other vendors detect this piece of malware and you don't. Go get your act together and release a new signature ASAP".
Let me just come out and say this right up front "The results from VirusTotal are a very poor indicator of the detection capabilities of a security product". Why you may ask ? Its simple. VirusTotal runs outdated and partial versions of the traditional AntiVirus engines. These are engines that look for patterns in the files and as you well know, are not very good at detecting 0-day polymorphic threats which just happen to be the vast majority of threats out there today. Fortunately for the user, most modern security products depend very little on their AV engines for proactive detection. Instead they have multiple engines (or layers as they are sometimes called) to make up for the lack of detection from these traditional string-scanning engines. Some of these include Reputation-based engines, IPS engines, Browser Protection engines, Buffer-overflow engines, Behavioral engines and other Heuristic engines. All of these engines are far more effectiveness at detecting the fresh batch of new malware that appears every single day. And yet, none of them are used by VirusTotal.
Don't believe me ? Try a simple test. Take a URL that you know is hosting malicious software or an exploit kit. Using IE with various vulnerable plugins like Java, Flash etc., browse to that URL. Here is what you are likely to see:
- URL Blacklisting Layer - the website will likely to be blocked by the URL blacklisting layer of your product.
- Vulnerability Protection Layer or IPS - If the above layer doesn't block it, then your networking scanning layer or IPS will inspect the traffic and block the traffic if in fact that URL is exploiting one of the dozens of vulnerabilities out there
- Shell-code Protection Layer - If the above layers don't block the attacker, then likely the product will detect the heap-corruption, heap-spraying and the shell-code executing and terminate the browser
- Reputation Layer - If the above layers don't block it, then the reputation of any created PE files will be looked up in the cloud and will likely block that PE file.
- AVE Layer - Here the traditional AVE engines scan the PE file for known AVE signatures and might block it. This layer has gotten a lot more sophisticated over the years, now employing heuristic techniques as well as emulation.
- Behavioral Layer - if you are able to run the file because the other above layers missed it, the behavioral layer will monitor the running exe and all the actions it performs on the machine and if it notices something suspicious, it will block it. Symantec's SONAR engine is the undisputed champion here.
- HIPS Layer - A lot of products have a trip-wire like HIPS layers that will alert the user on any modifications to sensitives poritions of the OS, like the hosts file etc. Personally I think these products should be avoided like the plague since they are highly False Positive prone.
- Phone-home IPS inspection layer - The last layer is an IPS layer that looks at outbound network traffic and will identify running malware that has bypassed all other layers and is now phoning home. It will detect this and either block the traffic rendering the malware useless or remove the malware altogether.
As you can see your typical Internet Security product is far more sophisticated than you might think, and none of these layers are used/tested by VirusTotal. Its also worth reiterating that VirusTotal doesn't even use all the engines within the AVE Layer. So its somewhat neutered even in the layer it does use.
Bottom line, don't use Virus Total as a tool to compare the effectivness of two security products. Even VirusTotal has a page that recommends not doing this. See https://www.virustotal.com/faq/. Specifically it says
Why do not you include statistics comparing antivirus performance?
At VirusTotal we are tired of repeating that the service was not designed as a tool to perform antivirus comparative analyses, but as a tool that checks suspicious samples with several antivirus solutions and helps antivirus labs by forwarding them the malware they fail to detect. Those who use VirusTotal to perform antivirus comparative analyses should know that they are making many implicit errors in their methodology, the most obvious being:- VirusTotal's antivirus engines are commandline versions, so depending on the product, they will not behave exactly the same as the desktop versions: for instance, desktop solutions may use techniques based on behavioural analysis and count with personal firewalls that may decrease entry points and mitigate propagation, etc.
- In VirusTotal desktop-oriented solutions coexist with perimeter-oriented solutions; heuristics in this latter group may be more aggressive and paranoid, since the impact of false positives is less visible in the perimeter. It is simply not fair to compare both groups.
- Some of the solutions included in VirusTotal are parametrized (in coherence with the developer company's desire) with a different heuristic/agressiveness level than the official end-user default configuration.
If you have a security pro on your staff that presents you with results of a virustotal scan to convince you that you should change your security product, FIRE HIM!
