Refactoring has been reported as a helpful technique to systematically improve non-functional attributes of software. This paper evaluates the relevance of refactoring for improving usability on web applications. We conducted an experiment with two replications at different locations, with subjects of different profiles. Objects chosen for the experiment were two e-commerce applications that exhibit common business processes in today's web usage. Through the experiment we found that half of the studied refactorings cause a significant improvement in usability. The rest of the refactorings required a post-hoc analysis in which we considered aspects like user expertise in the interaction with web applications or type of application. We conclude that, when improving quality in use, the success of the refactoring process depends on several factors, including the type of software system, context and users. We have analyzed all these aspects, which developers must consider for a better decision support at the time of prioritizing improvements and outweighing effort.Laboratorio de Investigación y Formación en Informática Avanzada