The surface topography of engineering rough surfaces has an enormous influence on contact mechanics of the interface and further the dynamics of the contact system. In this paper, the force-deflection characteristic and the nonlinear vibration of a rough surface interacting with a rigid flat surface are studied. A three-dimensional rough surface is constructed using a modified two-variable Weierstrass-Mandelbrot fractal function and the force-deflection is determined to be a positive power-law function. The power has the values larger than unity and increases with a rougher surface topography. Approximation of the force-deflection characteristic is also presented. Accordingly, the natural frequency is determined both exactly and approximately from the numerical calculation of the natural period and using the multiple scales method on the approximate equation, respectively. The primary resonance responses under harmonic excitation are also determined as well as the jump-up and jump-down responses. The nonlinear contact stiffness characteristics and the effect on natural frequency and the frequency responses are illustrated for different rough surface topographies. It is shown that the variation of natural frequency with amplitude and the multi-valued region of frequency responses increase with a rougher surface topography; however, the jump-up and jump-down frequencies both decrease, as well as the peak amplitude. (C) 2014 Elsevier Masson SAS. All rights reserved.Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP)