The simplex algorithm — Discussions in class Michel Bierlaire 1 Summary The simplex algorithm solves a linear optimization problem by visiting the vertices of the constraint polyhedron. It exploits the representation of vertices by feasible basic solutions, and follows the basic directions to move from one vertex to the next. 1.1 Main concepts • Basic directions, • reduced costs, • solution on a vertex, • from vertex to vertex, • the simplex tableau, B−1 A B−1 b cT − cTB B−1 A −cTB B−1 b • the initial tableau: the simple case, • the initial tableau: the general case. For the multiple choice questions below, the correct responses are in bold font. 1 2 Reduced costs Consider the condition “the reduced costs are all non negative”. Is it a sufficient/necessary optimality condition? • sufficient and necessary, • sufficient and not necessary, • not sufficient but necessary, • neither sufficient, neither necessary. 3 Degeneracy: example We propose an example where the optimal solution is degenerate, and the reduced costs are non negative. min x1 + x2 subject to x1 ≥ x2 , x1 ≥ 0, x2 ≥ 0. Standard form: min x1 + x2 subject to −x1 + x2 + x3 x1 x2 x3 = 0, ≥ 0, ≥ 0, ≥ 0. 1 A = (−1 1 1), b = 0, c = 1 . 0 Consider the feasible basic direction where x2 is in the basis, and calculate the reduced costs. 1. Is it an optimal vertex? Yes. If x2 is in the basis, we have B = B−1 = 1, and xB = B−1 b = 1 · 0 = 0. Therefore, all the variables are equal to zero. The value of the objective function at (0, 0) is zero. It cannot reach a lower value without violating the constraints. 2 2. Why is there a negative reduced cost? (a) Because the slack variable is out of the basis. (b) Because the basis is degenerate. It is false, because a basis can be degenerate, and have all the reduced costs non negative, as illustrated by the last example below. (c) Because it is a minimization problem. (d) Because the corresponding basic direction is infeasible. (e) Because there is only one basic variable. If x2 is in the basis, we have B = B−1 = 1. The basic entry of the basic direction that tries to put x3 in the basis is (d3 )B = −B−1 A3 = −(1)(1) = −1. Therefore, the direction is 0 0 d3 = (d3 )B = −1 1 1 and the reduced cost is c̄3 = c3 − cTB B−1 A3 = 0 − (1)(1)(1) = −1. Consequently, the objective function decreases along this direction. The basic entry of the basic direction that tries to put x1 in the basis is (d1 )B = −B−1 A1 = −(1)(−1) = 1. Therefore, the direction is 1 1 1 (d1 )B = d1 = 0 0 and the reduced cost is c̄1 = c1 − cTB B−1 A1 = 1 − (1)(1)(−1) = 2. Consequently, the objective function increases along this direction. 3 d1 d3 3. If, instead of minimizing the objective function, we maximize it on the same set of constraints, what happens? • As the polyhedron has only one vertex, it corresponds to the optimal solution of the maximization problem as well. • There is no optimal solution. • The optimal solution is not on a vertex. 4 Solution on a vertex 1. Consider a linear optimization problem, with at least one optimal solution. Which one of the following statements is correct? • There exists an optimal vertex of the constraints polyhedron. Indeed, this is Theorem 16.2, p.365. • An optimal solution is always at a vertex of the constraints polyhedron. No. It may happen that all solutions in a face of the polyhedron are optimal. In that case, solutions that are not vertices are also optimal. • There exists a vertex of the constraints polyhedron which is optimal only if the problem is written in standard form. No. The property is true irrespectively of how the polyhedron is written. Indeed, any polyhedron can be written in standard form, and the proof of Theorem 16.2 applies. 2. One of the following optimization problems has an optimal solution which is not a vertex. Which one? 4 (a) min x s.t. 0 ≤ x ≤ 1. The unique solution is x∗ = 0, which is on a vertex. (b) min x1 + x2 , s.t. x1 , x2 ≥ 0. The unique solution is x1 = x2 = 0, which is on a vertex. (c) min x1 + 2x2 s.t. x1 + 2x2 ≤ 4, x1 , x2 ≥ 0. The unique solution is x1 = x2 = 0, which is on a vertex. (d) max x1 + 2x2 s.t. x1 + 2x2 ≤ 4, x1 , x2 ≥ 0. Note that it is a maximization problem. All solutions such that x1 + 2x2 = 4 and x1 , x2 ≥ 0 are optimal. In particular, the solutions x1 = 0, x2 = 2 and x1 = 4, x2 = 0 are optimal, and are on a vertex of the polyhedron. But the solution x1 = 2, x2 = 1 is also optimal, and is not on a vertex. (e) max x s.t. 0 ≤ x ≤ 1. The unique solution is x = 1, which is on a vertex. (f) max x1 + x2 s.t. x1 , x2 ≥ 0. This problem is not bounded, and has no optimal solution. 5 Step along the direction 1. The length of the step α to perform along the basic direction dp is determined using exactly one of the following conditions. • The gradient at x + αdp is zero. No, the gradient is constant and always equal to c. Therefore, except for a trivial problem where c = 0, the gradient is never zero. • α = 1. No, the value one is arbitrary. • α is equal to the reduced cost associated with dp . No, the reduced cost is the slope, the directional derivative, of the objective function in the direction dp . • α corresponds to the first constraint that is activated. Yes, we follow the descent direction dp as far as we can, which is when the first constraint is met. Beyond, the iterate would become infeasible. • α corresponds to the last constraint that is activated. No, it must be the first, as discussed above. • There are m + 1 variables in the basis at x + αdp . No, there are always m variables in the basis. 5 • There are m − 1 variables in the basis at x + αd. No, there are always m variables in the basis. 2. Consider an optimal feasible basic solution, such that the reduced cost associated with the non basic variable xj is c̄j < 0. What is the length α of the step to perform along the basic direction dj ? (a) 0, (b) 1, (c) +∞, (d) this situation is not possible. The situation is possible, when the optimal basis is degenerate. As the solution is optimal, any descent direction must be infeasible. And dj is a descent direction, as the reduced cost is negative. Therefore, the step is zero. 3. Consider the problem max x1 x∈R2 subject to x1 + x2 = 0, x1 , x2 ≥ 0. Write the problem in standard form, and consider the feasible basic solution where x2 is in the basis. Which one of the following statements is correct? • The reduced cost is negative, and α = 0. • The reduced cost is negative and α = 1. • The reduced cost is negative and α = +∞. • The reduced cost is positive, and there is no need to calculate α. • The reduced cost is zero, and there is no need to calculate α. The problem in standard form is min −x1 x∈R2 6 subject to x1 + x2 = 0, x1 , x2 ≥ 0. The data of the problem is A = (1 1), b = 0, c = −1 0 . Note that the only feasible solution is x1 = 0, x2 = 0, as illustrated in the following figure. 1 x2 d2 0.5 x1 −1 −0.5 0.5 −0.5 1 d1 −1 If x2 is in the basis, we have x1 = 0. As B = 1, x2 = xB = B−1 b = 1·0 = 0. The basic direction d1 is such that (d1 )B = −B−1 A1 = −1 · 1 = −1, so that 1 . d1 = −1 The reduced cost is c̄1 = c1 − cTB B−1 A1 = −1. Note that, if we decide to put x1 in the basis, we have x2 = 0. As B = 1, x1 = xB = B−1 b = 1 · 0 = 0. The basic direction d2 is such that (d2 )B = −B−1 A1 = −1 · 1 = −1, so that −1 . d2 = 1 The reduced cost is c̄2 = c2 − cTB B−1 A2 = 0 − (−1)(1)(1) = 1. 7 6 Tableau Consider the following tableau. x1 x2 x3 x4 x5 x6 1 -1 0 -1 -1 2 1 0 -1 0 -1 1 1 0 0 0 1 0 2 6 -8 Only one of the following statements is incorrect. 1. x3 is non basic. Indeed, if it were basic, the corresponding column would contain only zero’s, except for one “one”. 2. x5 = 2. Indeed. x5 is in the basis, and the position of the “one” refers to the corresponding row. Therefore, we read its value at row one in the last column. 3. x2 = 0. Indeed, x2 is out of the basis. 4. The solution is not optimal. Indeed, the reduced cost of x3 is negative, so that the tableau is not optimal. It may happen that a tableau which is not optimal corresponds to an optimal solution. However, it is not the case here as the feasible basic solution is not degenerate. 5. The value of the objective function is −8. No. The value of the objective function is the value in the last row and the last column, with a different sign. The value of the objective function is 8. 6. x6 is in the basis. Indeed, the corresponding column contains only zero’s, and one “one”. 7. If x3 enters the basis and x5 exits the basis, the length of the step along the basic direction is α = 2. Indeed, the value of α5 is 2/1 = 2, and the value of α6 is +∞. The value of α is the smallest of the two, that is 2. 8. If x3 enters the basis, the basic direction is 0 0 1 d3 = 0 . −1 0 8 Indeed, the entries of the upper-left part of the tableau contain −dB . Therefore, −1 . (d3 )B = 0 As the variables in the basis are x5 and x6 , we have (d3 )5 = −1 and (d3 )6 = 0. As it is the third basic direction, we have (d3 )3 = 1. All the other entries are zero, so that 0 0 1 . d3 = 0 −1 0 7 The initial tableau At the end of phase I, we obtain the following tableau: x1 x2 x3 x4 xa1 xa2 xa3 xa4 1 0 0 0 0 0 1 0 0 0 0 0 0 1 0 1/2 −3/4 0 1/3 0 1/2 1/4 −1 0 2 −1/2 1/4 −1 0 2 0 0 1 0 0 1/2 −3/4 0 1/3 1 1 1/2 0 1/3 0 x1 x2 xa3 x3 What should be done next. 1. Pivot the tableau to take xa3 out of the basis. xa3 should indeed not stay in the basis, but there is no valid pivot. Indeed, the only original variable candidate to enter the basis is x4 , and the corresponding pivot is zero. 2. Stop the algorithm as the original problem is not feasible. No. As the objective function of the auxiliary problem is zero, we have found a feasible solution to the original problem: x1 = 1, x2 = 1/2, x3 = 1/3, x4 = 0. 3. Calculate the last row of the tableau for the original problem. No. The tableau must be “cleaned” first, in the sense that all auxiliary variables must be removed. 9 4. Remove the third row of the tableau. Indeed, as discussed above, there is no way to pivot xa3 out of the basis. It means that the corresponding constraint is redundant, and the row can be removed. 5. Remove the columns of the tableau corresponding to the auxiliary variables. No. This cannot be done while an auxiliary variable is in the basis. If time allows... A Reduced costs: example Consider the lame linear optimization problem as last week: max x1 subject to x1 ≤ 1, x2 ≤ 2, x1 , x2 ≥ 0. In standard form: Standard form: min −x1 subject to x1 + x3 = 1 x2 + x4 = 2 x1 , x2 , x3 , x4 ≥ 0 −1 0 1 1 0 1 0 , c= , b= A= 0 . 2 0 1 0 1 0 The feasible set is represented in the following picture: • • • • 10 Consider x1 and x4 in the basis: x1 = 1, x2 = 0, x3 = 0, x4 = 2. The reduced cost associated with x2 is • positive, • negative, • zero. The formula for the reduced costs is c̄j = cj − cTB B−1 Aj . where j is the index of a non basic variable. As 0 −1 , , and A2 = B = I, cB = 1 0 we have c̄2 = 0. From a geometrical point of view, the direction starts from the point (1, 0) and increases the value of x2 along the face. In this direction, the value of x1 does not change. Therefore, the value of the objective function does not change. And the directional derivative, that is the reduced cost, is zero. • (1, 0) A.1 Degeneracy: another example Consider the problem min x1 + x2 subject to x2 − x3 x1 x2 x3 11 = 0, ≥ 0, ≥ 0, ≥ 0. The data of the problem are 1 A = (0 1 − 1), b = 0, c = 1 . 0 We have three possible bases. 1. x1 in the basis: impossible, as B = 0 would not be invertible. 2. x2 in the basis: B = 1, cB = 1. x2 = xB = B−1 b = 1 · 0 = 0. All variables are equal to zero. The reduced costs are c̄1 = c1 − cTB B−1 A1 = 1 − (1)(1)(0) = 1, c̄3 = c3 − cTB B−1 A3 = 0 − (1)(1)(−1) = 1. They are both non negative. The sufficient condition for optimality is verified. 3. x3 in the basis: B = −1, cB = 0. x3 = xB = B−1 b = −1 · 0 = 0. All variables are equal to zero. This is the same vertex as the previous basis. The reduced costs are c̄1 = c1 − cTB B−1 A1 = c1 = 1, and c̄2 = c2 − cTB B−1 A2 = c2 = 1. They are both non negative. The sufficient condition for optimality is also verified. Note that the constraints x2 − x3 = 0 and x3 ≥ 0 are equivalent to the constraint x2 ≥ 0, and can be removed. Therefore, the problem can be written min x1 + x2 subject to x1 ≥ 0, x2 ≥ 0. In this case, the optimal solution is the same, but the feasible basic solution is not degenerate. 12