For the HMM exists defects in application in the aspect of software behavior prediction, namely, HMM could trap into local optimization because of the problem of B-parameter, which results in the decrease of HMM’s precision. This paper builds a new model HMM-ACO through combining Ant Colony Optimization (ACO) algorithm with HMM, with system calls as the data source, improving the prediction accuracy rate of HMM. In order to eliminate the HMM’s reflection on observations characteristics, this paper puts forward a new approach to recognize software behavior with hidden states.