We derive Virasoro constraints for the zero momentum part of the QCD-like partition functions in the sector of topological charge v. The constraints depend on the topological charge only through the combination N-f +betav/2 where the value of the Dyson index beta is determined by the reality type of the fermions. This duality between flavor and topology is inherited by the small-mass expansion of the partition function and all spectral sum rules of inverse powers of the eigenvalues of the Dirac operator. For the special case beta =2 but arbitrary topological charge the Virasoro constraints are solved uniquely by a generalized Kontsevich model with the potential V(X) = 1/X.