The aim of this work was to estimate the general and specific combining ability of peppers by measuring fruit quality and yield traits. This experiment was carried out on the garden field from Universidade Federal de Viçosa (UFV), Minas Gerais State—Brazil. Eight lines of Capsicum baccatum belonging to the UFV Horticultural Germplasm Bank were chosen based on their broad genetic and phenotypical background variation and then they were crossed in a complete diallel way. The F1 seeds of the 56 hybrids and eight parents were planted in the field in a randomized complete block design. The data were submitted to ANOVA and the means were grouped by Scott–Knott test (P ≤ 0.01). Significant variation for fruit quality and yield components was observed among parents and F1 generation. Analysis of variance for the combining ability showed that GCA effects exhibited significant difference and SCA effects of the crosses were significant, except for the height of first bifurcation. For almost all characters both additive and non-additive effects influenced the performance of hybrids.