In this era of “precision” medicine, image-guided intervention (IGI) enables real-time customized and accurate treatment using imaging phenotype-based approaches. Colorectal cancer (CC) is the third most commonly occurring cancer, resulting in nearly 10% of cases over the globe. Colorectal cancer classification (CCC) of histopathological images by artificial intelligence (AI) approaches not only enhances the accuracy and classifier results but also allows physicians to make prompt decisions. In this view, this article introduces a novel Galactic Swarm Optimization with Deep Transfer Learning Driven Colorectal Cancer Classification (GSODTL-C3M) model for IGI. The primary aim of the GSODTL-C3M model is to appropriately categorize the test images into the existence of CC. To accomplish this, the presented GSODTL-C3M model employs image pre-processing using the bilateral filtering (BF) technique to remove noise. Besides, Adam optimizer with the MobileNet model is applied as a feature extractor. Finally, the GSO methodology with long short-term memory (LSTM) is employed to recognize and classify CC. An extensive range of simulations was taken place, and the results stated the advanced performance of the current state of art classification methodologies.