A discrete group G of finite virtual cohomological dimension is called topologically rigid, if it admits a manifold classifying space which is unique up to G-homeomorphism. Examples of such groups include the torsion free groups that appear in the work of Farrell-Jones and Coxeter groups. Let D be a topologically rigid CAT(0)-group that acts on a Coxeter group W via Coxeter Graph automorphisms. We show that, under certain low dimensional restrictions, the semidirect product of W and D obtained from this action is topologically rigid. Examples of such groups include the Weyl Groups of finite parabolic subgroups of Coxeter groups.