Mobile technologies make their headway by offering more flexibility to end-users and improve the productivities. Within the application of ubiquitous access and pervasive communication, security (or privacy) and QoS (Quality of Service) are two critical factors during global mobility, so how to get a smooth and fast handover based on a user privacy protected infrastructure is our focus. Based on a user-centric vir-tual identity defined by EU IST project Daidalos, this paper firstly proposes an effective infrastructure which protects the context-driven access policies for online services in order to avoid attacks by malicious eaves-droppers. In the proposed infrastructure, SMAL and Diameter are used to securely protect and deliver au-thenticated and authorized entities and XACML is used to authorize the user-level privacy policy. On the basis of it, a dynamic fast authentication and authorization handover mechanism is proposed which can save one trip communication time consummation between administrative domains.