With the widespread deployment of biometric recognition,personal data security and privacy are attracted more and more attentions.A crucial privacy issue is how to ensure the security of user template.This paper proposes a novel template protection algorithm for face recognition based on chaotic map.Each face template is corresponding to different chaotic sequence produced by system master key and user identification number.The order of chaotic sequence controls the substitution index of face template.Experiment results on facial FERET database show that our algorithm can significantly improve the recognition performance and ensure the security of face template.