ON THE SOLUTIONS OF A STOCHASTIC CONTROL SYSTEM

This paper presents generalizations of the work in [1], [2] to include controlled stochastic processes which take values in a certain class of Fr6chet spaces. The crucial result is an extension of Girsanov’s technique for defining solutions of stochastic differential equations by an absolutely continuous transformation of measures. The result is used to prove existence results for stochastic control problems and for a class of two-person zero sum games.

1. Introduction and summary.Consider a controlled stochastic process represented by the stochastic differential equation dX(t) f (t, X, u(t, X)) dt + dB(t), [0, 1], where B(t) is a Fr6chet-valued Brownian motion, X(t) is the state process.The drift f, and the control u, depend at any time on the past of the state process, {X(s), s __< t}.For the case where the Fr6chet space is Rn, a satisfactory theory dealing with the problem of existence of solutions of the differential equation and existence of optimal control laws is now available [1], [2].A crucial building block in this theory consists in defining a solution of the differential equation via an absolutely continuous transformation of measures.Each control thereby de- fines a solution characterized by its (unique) probability law which is absolutely continuous with respect to Wiener measure.Thus the influence of a control law upon the system is captured in the Radon-Nikodym derivative of the resulting probability law with respect to Wiener measure.Questions dealing with the existence of an optimal control can then be converted into questions about the compactness (in an appropriate sense) of the set of Radon-Nikodym derivatives.
The measure transformation technique mentioned above is due originally to Girsanov 16].
This paper deals with these same questions for the case where the state space is infinite-dimensional.The problem of characterizing Brownian motion in infinite-dimensional spaces is a difficult one and has been resolved for certain Fr6chet spaces only.This is described in the next section, where some additional properties of such Brownian motion as sample continuity and stochastic inte- gration are established also..In 3 the result of Girsanov is extended to cover the differential equation under consideration.Once this has been achieved the tech- niques of [1], [2] apply without change and the existence of optimal controls and latter are generalizations of the work of Gross [4].While the generality of the immediately following discussion is not used subsequently, it does serve to indicate some directions along which the results reported here can be further pursued.Some notation and definitions are introduced first.
For a locally convex Huasdorff topological (real) vector space X let X* denote its topological dual.A dual system or duality over the reals consists of two vector spaces X, Y and a bilinear form (.,.X Y--* R that separates points for both X and Y.For a pair X, Y in duality, let FD(X) denote the collection of all finite-dimensional subspaces of X.For G c X, let///(Y, G) be the a-algebra of Y generated by the sets (y: (x, y) B}, where x G, B is a Borel set in R. Thus /J/(Y, G) is the smallest r-algebra on Y for which each x G, regarded as a function on Y, is measurable.Finally let (g(Y, X) U (/J/(Y, G)IG FD(x).A cylinder set measure on Y is any nonnegative, finitely additive set function m on (g(Y, X) with m(Y) such that m is countably additive on '(Y, G) for each G FD(X).The following notion of a measurable seminorm given in [3] is crucial to the analysis.
DEFINITION 1.Given a duality X, Y, (.,. , and a cylinder set measure m on Y, a seminorm I" on Y is said to be m-measurable if for each e > 0 there is a G FD(Y) such that if F FD(X) and F d_ G (i.e., (x, y 0 for x e F, y G), then or equivalently m{y:ly-F+/-I _-< e} _-> 1-e m*{y:iy-Gl<e} l-e, where m* is the outer measure on cg(y, X) induced by m.
FUNDAMENTAL THEOREM.Let I" be a Mackey-continuous seminorm on Y and suppose that it is m-measurable.Then the cylinder set measure induced by m on the Banach space Y/I" obtained from Y via the seminorm ]. extends to a regular Borel measure.
From here on the discussion is specialized to a fixed, separable Hilbert space H.It is assumed that there is given for each R+ a cylinder set measure Pt on H such that pt is a canonical normal distribution on H with variance param- eter (see [4] or [5]).It is also assumed that there is given an increasing family of Mackey-continuous seminorms I" I, j 1, 2,..., on H. Let F be the Fr6chet space obtained from H with respect to the topology defined by the seminorms I" I by completion modulo the intersection of their null spaces (which without loss of generality is assumed to equal {0}).As a corollary to the Fundamental Theorem it is proved in [3, Cor.2.1] that each Pt, tR+, extends to a regular Borel measure, denoted #t, on F. For future reference note that H c F, F* H* This means that if C e (H, is of the form C P-(E), where P is the orthogonal projection of H onto a subspace L FD(H) and E is a Borel subset of L, then lxl dx, pt(C) (2t)-"/Z feexp (where n is the dimension of L, and ]. lu denotes the norm on H induced by its inner product.H, and define the maps i'H F, j'F* H*, as the canonical injections.Finally let N(F) denote the Borel sets of F. Throughout, H and F denote the spaces introduced here.
The collection #t, e R/, will be used now to define a Brownian motion with values in F. For eacl e R/ let F be a copy of F. For each finite collection t < < t,, let tl,'",trt be the measure on (]-[7= Ft" HT=l (Ft')) defined by fAn d#tl'""t" d#,._t._,(x,,) dl.tt=_t,(x2) dYt,(Xl) for Ai (F"), 1, ..., n.Since sets of the form l--I'= Ai generate the product r-algebra ]-I'-1 (F") the measure #,1,...,,, is defined.To show that the family of measures #,1,...,,, is a projective system, it is necessary to verify consistency.Since   F is separable in the topology determined by the countable seminorms I. j, the measure #,1,...,,, is determined by sets of the form Ai (I P)F @ Fi, where P is an orthogonal projection with finite dimensional range and F is a Borel subset of PF.Let A1, "'", A, be such sets, and suppose Aj F. Then fA fA d'''''"= fA fA dl'tt"-t"-'(xn)'"d#t*(x1) 2--Xl y.-1Xi @,._,._ ,(x,) dp,._,._ ..dp,,+ =-t,+ dpt,+ ,-t,_,(x) [l,',[j l[j+ 1,",n j-1 j+l In the above p is the measure of a Brownian motion with values in the finite- dimensional space PF and so the third equality above follows from the Markov property of such a process.Since each of the measures #,, R/, is a regular Borel measure, the projective system of measures admits a projective limit [6, p. 49].The projective limit thus obtained is denoted (f2, ff, P).Evidently, (f2, )= 1-[,e+ (U, (F')).It will be assumed that (f2, if, P) is complete.For each let X denote the F-valued evaluation map on f2 at t. Let o be the smallest completed a-algebra with respect to which X is measurable for s =< t.Then (Xt, t, P)te is a Brownian motion as defined below.DEFINITION 2. A stochastic process (Xt, ,, P)tzR+ (or simply (X,, P)tzR or (X,) if there is no ambiguity) is said to be a Brownian motion with values in the Frdchet space F induced from a family of canonical normal distributions on a Hilbert space H, if for each e F* (the topological dual of F) the real-valued process ((1, X,), ,P)tzR+ is a real-valued Brownian motion with E<l, Xt> 2 tljll2n.(Here and throughout I" In denotes the norm on H and j'F* --, H* is the canonical injection mentioned before.) The process (Y,, P),R is said to be a modification of the process (X,, P),R if for each R+X,(og) Y,(o) a.s.(the null set {X, # Y,} may depend on t).Downloaded 09/10/14 to 129.237.46.100.Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.phpFor Fr6chet-valued Brownian motion there is a modification which has continuous sample paths as shown by the following lemma.
LEMMA 1.Let (X t, , P)tR +be a Frchet-valued Brownian motion.There is a modification of Xt with continuous sample paths.
Proof.Since a countable number of seminorms determine the topology of a Fr6chet space it suffices to verify the continuity of the sample function of Brownian motion with respect to each of these countable number of seminorms and there- fore it is enough to prove the lemma for a Banach-valued Brownian motion.
Fernique [7] has shown that for a Gaussian random variable X on a topo- logical vector space with a measurable seminorm I. there is an > 0 such that (1) Combining (1) with a result of Nelson [8, Thm.2] as used by Gross [9, p. 134] it follows that the Banach valued Brownian motion has a modification with con- tinuous sample paths.
Since stochastic integrals will be used subsequently, a family of processes has to be described that can serve as integrands.The following definition gives such a family.DEFINITION 3. Let / be a separable Hilbert space.An/-valued stochastic process ()g on (f, , P) that is adapted to ()tR is said to be predictable if the map (og)'R+ f-/ is measurable with respect to the a-algebra on R+ f generated by the left-continuous/-valued processes adapted to ()R Real-valued stochastic integrals will now be defined from F-valued Brownian motion and predictable H-valued processes.LEMMA 2. Let (d/)tR be a predictable H-valued process with (2) E I,lr dt < , and let (Bt,tt, P)teR be an F-valued Brownian motion.Then the real-valued process Yt, , P Y (Os, dBs), is a square integrable martingale which has a modification with continuous sample paths.(The integral (3) is defined in the course of the proof.) Proof.Let e, > 0 be a sequence decreasing to zero and for each n let Pc. be a projection on H with finite-dimensional range P.H c jF* such that Let 11, "", lu be an orthonormal basis for P.H.The process (P,) defined by (4) Downloaded 09/10/14 to 129.237.46.100.Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php is a (continuous) martingale since by the definition of the process (B), ((/i, B)) is a real-valued Brownian motion.By Doob's inequality [10, p. 353] the sequence of continuous martingales (Y(P,)) converges uniformly on compact subsets of R+ because EIY,(P,) Y(P)I 2 E I(P, P,,)&I ds which vanishes as m, n increase to infinity.The integral (3) is defined as the limit of the martingales Y(P.).Clearly the limit does not depend on the particular choice of the projections P. and Y is evidently square integrable.U COROLLARY 1.Let ()R be a predictable H-valued process with (5) 112 ds < o a.s.for all t.
Then the real-valued process Z defined by is a locally square integrable martingale which has a modification with continuous sample paths. 2  The following representation of square integrable functionals on the F-valued Brownjan motion probability space will be useful subsequently.For R"-valued Brownian motion, K. It6 [11] has obtained this representation by describing results of Wiener [12] and Cameron-Martin [13] in terms of stochastic integrals.
PROPOSITION 1.Let (g), ,, P) be the probability space for an F-valued Brownian motion (B, ')tR /.Let f be a real-valued square integrable functional on (f, if, P).Then f can be represented as (7) f= c + (,dB), where c Ef and (lt)te R is a predictable H-valued process with e Itl2ndt < .
Proof.Since H is separable so is F, hence by the Hahn-Banach theorem, there is a countable family F c F* that separates points of F. Consider the random variables (7,j' dBs), where 7 e F, e R+ and let be the algebra of real-valued random variables formed from these and the constant random variables.Since the random variables (7, j' dBs) are jointly Gaussian, it follows that c L2(p) and furthermore

L2(p)
mt is said to be a locally square integrable martingale if there exists a sequence of stopping times T. T a.s.such that mt r. is a square integrable martingale for each n.
Let f L2(p).There exists then a sequence g, in such that Elf g,I 2 O.
By properties of finite-dimensional Brownian motion, g, has a representation of the form (7) (see [15]), i.e., g. c. + <', dB>, where c, Eg, and (') is a predictable process with values in some finite- dimensional subspace L of H and with L c iF*.Since LZ(P)-convergence implies Ll(P)-convergence it follows that c, converges to c. Furthermore the stochastic integrals in (8) must be Cauchy; hence ElO' O'12n dt 0 as m, n . Since the sequence of processes (') are predictable, there must exist a predictable process (fit) such that and evidently (7) is satisfied.
3. Transformation of measures.Theorem below describes how Fr6chetvalued Brownian motion is transformed by changing the probability measure by an absolutely continuous substitution of the measure.This result was first estab- lished by Girsanov  [16] for the case of R"-valued Brownian motion.The result has been presented in [17] and a related result is given in [18].THEOREM-1.Let (Bt,,P)tO,ll be an F-valued Brownian motion and let (tt)tto, be a predictable H-valued process such that | IGlat<oo a.s,P, (9) d 0 Define the nonnegative process (Mr, , P),to,l by (10) M exp (d/z, dB> -} IGI2 Proof.Define the increasing sequence of stopping times T, by inf{tlM > n}, T, if the set above is empty. Because of (9) M has (a modification with) continuous sample paths, so T, ]" a.s.P, and in particular, (13) lim M t^r.Mt a.s.P.
The theorem will be proved once it is shown that (16) (N,, t, P) is a martingale, /(N N[)= [/[(t-s), where/ denotes expectation with respect to the measure P. Now to prove (16) it is sufficient to show instead that (18) (M,N,, , P) is a martingale.
Because suppose that (18) is true.Then using the fact that (M,, P) is a martingale, (N,[) E(M,N,J) (by [20, p.  Applying the differentiation formula for continuous martingales 15] gives
Substitution of this into (19) gives /(N 2 N2])= and so ( 17) is proved, l-I Ill(ts), In many applications of the transformation of measures technique ofTheorem the crucial difficulty is to verify that/3(f) 1.The next result gives a sufficient condition for/() 1.It is due to V. Beneg.
LEMMA 3 (Beneg).Let (Bt, t, P)tto,ll be an F-valued Brownian motion.Let 0Pt)t[o,ll be a predictable H-valued process such that ( 20) where I" is a seminorm on F and K, K' are constants.Then there is > and M < o, depending only on K and K', such that (21) E exp a (q, dBs) -]d/si2n ds < M.
In particular, E exp <Ps, dBs> -Iq,lH ds 1. Downloaded 09/10/14 to 129.237.46.100.Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.phpand so (21) follows by Fatou's lemma.The final assertion is then immediate because (21)implies that exp [ (ks, dBs 1/2'o I1 ds is a martingale, l-] 4. Preliminaries for optimization.The system to be controlled is represented by the stochastic differential equation ( 22) where B is an F-valued Brownian motion, X e F is the state with X o 0 a.s., and u is the control law taking values in a prespecified compact subset U F called the control set.The functionftakes values in H and :H F is the canonical injection.The first difficulty to be resolved is to define the solution of the differential equation ( 22) for a large class of control laws.This is achieved in the following manner.One starts with a process (X, , Po)to, which is an F-valued Brownian motion.For a given control law u an F-valued process B' is defined by B' X, if(s, X, u(s, X)) ds.
Next the probability measure Po is replaced by another probability measure pu B is an F-valued Brownian motion.The such that the process process (X,, , W),to,1 is then regarded as the solution of (22) corresponding to the control law u.To make this procedure precise the following notations and definitions are useful.DEFINITION 4. (a) is the linear space of all F-valued continuous functions, denoted by z, on [0, 1].
(b) For [0, 1], is the smallest a-algebra of subsets of cg which contain all sets of the form {z e lz(s) A}, where s [0, t] and A is a (topological) Borel subset of F. 6e .
Throughout the remainder of this paper f is a fixed probability space with an increasing family of a-algebras , [0, 1].ff .It will be necessary to consider different probability measures on the space (fL if).If Y, is a family of measurable functions on (f, ) and if P is a probability measure on (fL if), the stochastic process corresponding to P and the family Y will be denoted by (Y, , P)to,.Then the same family Y generates different stochastic processes corresponding to different probability measures.Finally let Po be a distinguished probability measure, and let X be a distinguished family of F-valued measurable functions on (fL ), [0, 1], such that the process (Xt, , Po)tto, is an F- valued Brownian motion with continuous sample paths.Unless mentioned other- wise the process X refers to this process.Also E o will denote expectation with respect to Po.The measure induced on and will be called the Wiener measure on (, 6).
The following conditions are imposed on the function f in (22).fl.f is a map from [0, 1] cg U into H and f is measurable with respect to the product a-algebra (R) (R) u, where (') is the family of Borel subsets of [0, 3 (u).
LEMMA 5. Let d? O. Let (Yt, , P)ttO,Xl be any process with continuous sample paths such that the stochastic process (Bt, , P)tto,xl defined by dB dY ice(t, Y)dt is an F-valued Brownian motion.Then exp (dp(s, Y), dBs) -Ib(s, Y)12u ds dP 1.
Remark.This corollary implies that the solutions of (22) are unique in a weak sense, i.e., all solutions of (22) which have continuous sample paths must induce the same measure on ((, 5).The qualification "weak" is inserted because only the uniqueness of the probability law has been proved.
Recall that (Xt, t, Po)tto, 11 is an F-valued Brownian motion with continuous sample paths.Also recall Definition 8. DEFINITION 9.For any subset A c tI) define (A) c L l(fl, if, Po) by @(A) {exp (b)lb e A}.
The corollary to Lemma 3 implies the next assertion.Proof.Let bl, b2, be a sequence from tI)" and let p be such that (25) lim Eol p exp (b.)l / 0, and Pt Eo{Plt}, [0, 1].By taking modifications if necessary it can be assumed that the martingales Pt and t(b,) have continuous sample paths so that, by Doob's inequality [10, p. 353], it follows from (25) that (28) Pt lim exp (t(b.)uniformly on [0, 1] a.s.Po.
Thus there is a causal map b e q)" such that th(t, X) lim b,(t, X) a.s. (R) Po and evidently p exp ((b).
The proofs of the next two results are identical respectively with the proofs of [2, Lemma 4] and [2, Thin.2] with some obvious notational changes.Hence the proofs are omitted.LEMMA 7. (") is a convex subset of L2(fL , Po).Then (o:) is a closed, convex subset of L l(fL ._,Po). 5. Applications.The results developed above immediately imply the existence of optimal control laws for a broad class of problems.Consider the control system dX(t) if(t, X, u(t, X)) dt + dB(t), e [0, 1], with X(0) 0 a.s.Suppose that f satisfies the assumptions fl to f5 and in addition the function fo in f4 satisfies assumption f6.f6.There exists K, K' such that fo(n) <= K + K'n for all n e R +.
In a manner corresponding exactly to the argument developed in [2], the results presented above can be used to obtain the existence of a saddle point for a class of two-person zero-sum games.Since there is nothing new here the details are omitted.
As an example of the class of the Brownian motions described here let (B,,0,,)to, 112 be a biadditive Gaussian process, i.e., a zero mean Gaussian process with independent increments in each coordinate of the index set such that E[B(tl,Z)B(t2, "/72)]--(/71 A tz)(Z A q72).
This stochastic process has continuous sample paths and when it is considered as indexed by one coordinate of the index set, it is a Brownian motion with values in the Banach space of continuous functions, C[0.1].Optimal control results can be obtained by the previous results for stochastic differential equations with this Brownian motion.
Remark on [2].It is necessary to make the assumption of uniform integrability of the family of densities to obtain the results in Theorems 4 and 5 of [2].One sufficient condition for this uniform integrability is that the growth of the drift term is at most linear.This fact is shown by Bene [1] by verifying that the family of densities have a bounded th.moment for some > 1.