||One of the most important assets to be protected is information, as every aspect of the life of a society deeply depends on the available information. Nowadays, information is stored, processed, and communicated by computers. It turns out that computers represent the most critical tool in modern society. A number of protection mechanisms are available so far, such as antivirus software tools, and biometric access control systems. For their effectiveness, frequent updates are needed, due to the rapid evolution of attack patterns. In fact, attacks are often devised and spread by running computer programs, which can produce new effective attacks in a short time frame. It turns out that machine learning techniques with their generalization capability are one of the favorite approaches to deploy protection and attack detection mechanisms. In this paper, we discuss the approaches that should be followed when devising machine learning techniques for security applications. In particular, we will focus on testing methodologies, performance measures, and techniques aimed at reducing the intrinsic variability of performance that often machine learning application exhibit in real-world scenarios.