Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP)Diabetic retinopathy (DR) is the leading cause of blindness in adults, but can be managed if detected early. Automated DR screening helps by indicating which patients should be referred to the doctor. However, current techniques of automated screening still depend too much on the detection of individual lesions. In this study, we bypass lesion detection, and directly train a classifier for DR referral. Additional novelties are the use of state-of-the-art mid-level features for the retinal images: BossaNova and Fisher Vector. Those features extend the classical Bags of Visual Words and greatly improve the accuracy of complex classification tasks. The proposed technique for direct referral is promising, achieving an area under the curve of 96.4%, thus, reducing the classification error by almost 40% over the current state of the art, held by lesion-based techniques.211193200Microsoft ResearchSao Paulo Research Foundation (Fapesp) [MSR-Fapesp 2008/54443-2, Fapesp 2010/05647-4]Amazon Web ServicesSamsung Electronics of AmazonFundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP)