The NIST Privacy Framework describes itself as a comprehensive approach to organization-wide privacy program management. However, inferences can yield sensitive information of identities or attributes from nonsensitive information. Privacy governance must protect this information. Although many people and organizations are expanding their privacy definitions to include inferences, our gap analysis reveals that the framework's mapped controls are insufficient for managing inference-driven risk. The framework does not attend organizational focus to privacy inference risk sufficiently to support its stated claim of comprehensive risk management. Applying the framework to past incidents where ostensibly protected information was re-inferred, we analyze how organizations can better mitigate inference-based privacy violations. Finally, we recommend detailed improvements to the framework's controls to account better for inferences. Our recommendations encompass augmenting and mapping additional privacy risk controls to increase implementing organizations' awareness of inference risks, updating controls that depend on protecting specific PII categories, and enhancing organizations' proficiency in translating legal and policy requirements into technical implementations.