Data envelopment analysis (DEA) has been widely applied in evaluating multicriteria decision making problems, which have multi-input and multi-output. However, the traditional DEA method does neither take the decision maker’s subjective preferences to the individual criteria into consideration nor rank the selected options or decision making units (DMUs). On the other hand, Satty’s analytical hierarchy process (AHP) was established to rank options or DMUs under multi-input and multi-output through pairwise comparisons. However, in most cases, the AHP pairwise comparison method is not perfectly consistent, which may give rise to confusions in determining the appropriate priorities of each criterion to be considered. The inconsistency implicates the fuzziness in generating the relative important weight for each criterion. In this paper, a novel method, which employs both DEA and AHP methods, is proposed to evaluate the overall performance of suppliers’ involvement in the production of a manufacturing company. This method has been developed through modifying the DEA method into a weighting constrained DEA method by using a piecewise triangular weighting fuzzy set, which is generated from the inconsistent AHP comparisons. A bias tolerance ratio (BTR) is introduced to represent the varying but restrained weighting values of each criterion. Accordingly, the BTR provides the decision maker a controllable parameter by tightening or loosening the range of the weighting values in evaluating the overall performance of available suppliers, which in hence, overcomes the two weaknesses of the traditional DEA method.