Since the seminal paper of Shapley, the theory of stochastic games has been developed in many different directions. However, there has been practically no work on the interplay between stochastic games and cooperative game theory. Our purpose here is to make a first step in this direction. We show that the Harsanyi-Shapley-Nash cooperative solution to one-shot strategic games can be extended to stochastic games. While this extension applies to general n-person stochastic games, it does not rely on Nash equilibrium analysis in such games. Rather, it only makes use of minmax analysis in two-person (zero-sum) stochastic games. This will become clear in the sequel.