In this paper, we discuss security of public-key cryptographic primitives in the case that the public key is fixed. In the standard argument, security of cryptographic primitives are evaluated by estimating the average probability of being successfully attacked where keys are treated as random variables. In contrast to this, in practice, a user is mostly interested in the security under his specific public key, which has been already fixed. However, it is obvious that such security cannot be mathematically guaranteed because for any given public key, there always potentially exists an adversary, which breaks its security. Therefore, the best what we can do is just to use a public key such that its effective adversary is not likely to be constructed in the real life and, thus, it is desired to provide a method for evaluating this possibility. The motivation of this work is to investigate (in)feasibility of predicting whether for a given fixed public key, its successful adversary will actually appear in the real life or not. As our main result, we prove that for any digital signature scheme or public key encryption scheme, it is impossible to reduce any fixed key adversary in any weaker security notion than the de facto ones (i.e., existential unforgery against adaptive chosen message attacks or indistinguishability against adaptive chosen ciphertext attacks) to fixed key adversaries in the de facto security notion in a black-box manner. This result means that, for example, for any digital signature scheme, impossibility of extracting the secret key from a fixed public key will never imply existential unforgery against chosen message attacks under the same key as long as we consider only black-box analysis.