SERRE PROBLEM FOR UNBOUNDED PSEUDOCONVEX Reinhardt Domains In C²

We give a characterization of non-hyperbolic pseudoconvex Reinhardt domains in ℂ2 for which the answer to the Serre problem is positive. Moreover, all non-hyperbolic pseudoconvex Reinhardt domains in ℂ2 with non-compact automorphism group are explicitly described.


-D is algebraically equivalent to a bounded domain; in other words, there is an
A ∈ Z n×n with | det A| = 1 such that ϕ A (D) is bounded and (ϕ A )| D is a biholomorphism onto the image. -logD contains no straight lines, and D ∩ (C j −1 × {0} × C n−j ) is either empty or hyperbolic (viewed as domains in C n−1 ), j = 1, . . . , n.
Definition 6 A domain D α,r − ,r + is said to be of rational type if tα ∈ (Q 2 ) * for some t > 0. Otherwise, D α,r − ,r + ⊂ C 2 is said to be of irrational type.
Proof We will show that any domain of the form (a), (b), (c) with rational β is algebraically equivalent to one of the domains enumerated above. Domains of the form D 0 , D * 0 , and D 0,r are in (d). Note that domains of the form (e) may be written as D q p . Moreover, domains of the form (f) are algebraically equivalent to D − q p . Therefore, any domain of the form (a) with β = 0 is of the form (e) or (f).
Fix a rational β > 0 and assume that β = p/q, where p and q are relatively prime natural numbers. There are m, n ∈ Z such that pm + qn = 1. Observe that the map- Similarly, one may show that D β,r is algebraically equivalent to A(r) × C * . What remains to do is to observe that the set {(z, w) ∈ C * × C: |z||w| < 1} is algebraically equivalent to D × C * .
Our paper is organized as follows. In Sect. 3 we give the proof of Theorem 1 for Reinhardt domains of the form (2) of the rational type. Section 4 is devoted to solving the Serre problem for Reinhardt domains of the form (2) of irrational type. We obtain there a natural correspondence between the automorphisms of Reinhardt domains of irrational type and the famous Pell's equation.
It seems probable to us that the solution of the Serre problem for C × C * is known, however, we could not find it in the literature. Therefore, in Sect. 6 we present ideas explaining how to easily extend the procedure used in [3] to this domain.
Finally, in Sect. 7, using the obtained results we present the proofs of Theorems 1 and 3.
The two following theorems providing us with some classes of domains for which the answer to the Serre problem is positive will be useful. The first of them is the so-called Stehlé criterion.
Theorem 8 (See [14,22]) Let D be a domain in C n . If there exists a real-valued plurisubharmonic exhaustion function u on D such that u • f − u is bounded from above for any f ∈ Aut(D), then D ∈ S.
Observe that the Stehlé criterion is satisfied among others by domains whose group of automorphisms is compact.
Let us recall basic facts and definitions related to holomorphic fiber bundles which will be useful in the sequel. For more information we refer the reader to [6].
Let E be an arbitrary holomorphic fiber bundle with the fiber X. The automorphisms of the form where τ α , τ β are trivializations (with associated domains α × X, β × X, respectively) are so-called transition functions. It is clear that On the other hand, having domains ( α ) α and a family of functions (τ α,β ) α,β ∈ Aut(( α ∩ β ) × X) satisfying the condition (3) we may define a holomorphic fiber bundle with the base B = α α and the fiber X by putting gluing the charts α × X via identification: Throughout this paper all holomorphic fiber bundles are understood to be of the form (4).

Rational Case
We start this section by collecting the formulas for automorphisms of Reinhardt domains of rational type. Some of them may be found in [9] and [21]. They may also be derived from the description of proper holomorphic mappings between Reinhardt domains of rational type obtained in [4] and [10]. As mentioned in Remark 7, any rational Reinhardt domain is algebraically equivalent to a domain of the form (d), (e), or (f).
In the case when C = C the mapping f is of the form where b ∈ O * (D) and = ±1.
The following lemma due to Stehlé will be needed.
Theorem 12 Any Reinhardt domain D in C 2 of the form (2) of rational type belongs to the class S.

Remark 13
Note that the function satisfying the Stehlé criterion does not exist for any domain appearing in the previous theorem. As an example, let us take D × C. It is sufficient to observe that for any f ∈ O(D, C * ) the mapping is an automorphism of D × C, and the growth of the function f may be arbitrarily fast.

Proof of Theorem 12
The proof is divided into four cases.
(a) First we focus our attention on the case when the fiber is equal to D = {(z, w) ∈ C 2 : |z||w| < 1}.
We will modify the idea used by Stehlé in the proof of Theorem 8. Let π : E → B be a holomorphic fiber bundle with a Stein base B and a fiber D. As we have already mentioned, E is assumed to be given by the formula (4).
Using the description of the group of automorphisms of the domain D we get that for any b ∈ α ∩ β a transition function τ α,β is of one of two following forms: Since τ α,β is holomorphic it follows that the mapping α ∩ β b → e iθ α,β (b) ∈ ∂D is holomorphic and therefore locally constant.
Letπ :Ẽ → B denote a holomorphic fiber bundle with the base B and the fiber D whose transition functions are defined byτ α, It follows from Theorem 9 thatẼ is Stein. Applying Lemma 8 to the family {π −1 ( α )} we get a locally finite covering {B j } and a family of strictly plurisubharmonic continuous functions {h j } satisfying conditions (i)-(iii) of Lemma 11. Let be a natural surjection between E andẼ.
For j ∈ N choose α j such that B j is relatively compact inπ −1 ( α j ). Let τ j := τ α j denote any trivialization of the fiber bundle π : E → B defined on p −1 (π −1 ( α j )). Put The choice of {B j } and a standard compactness argument guarantee that for any Put Condition (5) allows us to choose constants d ∈ (0, 1) and M > 0 such that Definẽ we obtain a plurisubharmonic continuous function defined on Let us definẽ Exactly as before, we check that v 3 ∈ PSH(p −1 (C 3 )).
Repeating this procedure inductively one may obtain a sequence of Since the covering {B j } is locally finite, putting v = lim j v j we define properly a plurisubharmonic function on E.
Letũ be a strictly plurisubharmonic continuous exhaustion function onẼ. It follows from the construction of the function v that is a plurisubharmonic continuous exhaustion function on E. Now, in order to show the Steinness of E it suffices to repeat the argument used by N. Mok in the proof of the improvement of the Stehlé criterion (see [14], Appendix III).
(b) Let us consider the case when the fiber D is of the form D = {(z, w) ∈ C 2 : |z| p |w| q < 1} for some natural relatively prime p, q, (p, q) = (1, 1).
Let π : E → B be a holomorphic fiber bundle with the fiber D. Using Theorem 10 we infer that the transition functions are of the form where a α,β ∈ O * (( α ∩ β ) × D) and θ α,β (b) ∈ R. Moreover, it may be shown that e iθ α,β is holomorphic and therefore locally constant. Letπ :Ẽ → B be a holomorphic fiber bundle with a fiber equal toD = {(z, w) ∈ C 2 : |z||w| < 1} and whose transition functions are defined in the following way: A direct computation allows us to observe that is a well defined proper holomorphic mapping. Therefore, by a result of Narasimhan in [15] (see also [5]) the manifold E is Stein if and only ifẼ is Stein. However, the Steinness ofẼ follows from the previous case.
Suppose that E is a holomorphic fiber bundle with the fiber D × C and the Stein base = α α . By Theorem 10, any transition function of E is of the form It follows from Theorem 10 that m α,β (x, ·) ∈ Aut(D) for any x ∈ α ∩ β (it may be shown that m α,β (x, ·) does not depend on x on connected components of α ∩ β ). LetẼ be a holomorphic fiber bundle with the base and the fiber D, whose transition functionsτ α,β ∈ Aut(( α ∩ β ) × D) are given by the formulas By Theorem 9,Ẽ is Stein. Now it is sufficient to observe that forms a holomorphic fiber bundle with the baseẼ and the fiber equal to C. Using again Theorem 9 we get the Steinness of the bundle E.
(d) To finish the proof of the theorem, it suffices to show that D = {(z, w) ∈ C * × C: |z p ||w q | < 1} ∈ S, where p, q ≥ 1 are relatively prime natural numbers, (p, q) = (1, 1). We proceed similarly as in case (b). Namely, once again we aim at reducing the situation to the already solved case (c).
Suppose that π : E → B is a holomorphic fiber bundle. Then its transition functions must be of the form for some a α,β ∈ O * (( α ∩ β ) × D) and θ α,β (b) ∈ R. It is seen that e iθ α,β may be chosen to be constant on connected components of α ∩ β . Therefore, putting we obtain a holomorphic fiber bundleẼ with the base and the fiber D × C * such that a mapping given by the formula is proper and holomorphic. The argument used in the proof of (b) finishes the proof of this case.

Irrational Case
The geometry of Reinhardt domains has been investigated in several papers (see [18,19] and references contained there).
In [10] the author obtained a complete characterization of proper holomorphic mappings between Reinhardt domains of irrational type. This characterization is of key importance for our considerations. Therefore, in Theorem 14 we collect the most crucial part of it.
Using Theorem 14 we shall obtain in the proof of Theorem 15 a full description of the group of automorphisms of Reinhardt domains of irrational type. As mentioned before, it turns out that this problem is connected with the famous Pell's equation.
Recall that any Reinhardt domain of irrational type is equivalent to a domain of the type (a), (b), or (c). Theorem 14 ([10], see also [19]) (i) (a) If α ∈ R \ Q, then the set of proper holomorphic mappings between D α,r and D β,R is non-empty if and only if log R log r ∈ Z + βZ and α log R log r ∈ Z + βZ.
(b) Let α, β ∈ R \ Q, and r, R > 1 be such that log R log r = k 1 + l 1 β and α log R log r = k 2 + l 2 β for some integers k i , l i , i = 1, 2. Then any proper holomorphic mapping f : D α,r → D β,R is one of the two following forms: where a, b ∈ C satisfy the equality |a||b| β = 1.
(a) If α > 0, β > 0, then the set Prop(D α , D β ) 3 is non-empty if and only if α = pβ for some p ∈ Q >0 . In this case, all proper holomorphic mappings between D α and D β are of the form where k, l are positive integers such that p = l k . (b) If α < 0, β < 0, then the set Prop(D α , D β ) is non-empty if and only if α = p 1 + p 2 β for some rational p 1 , p 2 , p 2 = 0. In this case, proper holomorphic mappings between D α and D β are of the form where k 1 , k 2 , l, k 1 > 0, are integers such that p 1 = k 2 k 1 , i p 2 = l k 1 . (c) If αβ < 0, then there is no proper holomorphic mapping between D α and D β .
Our aim in this part of the paper is to show the following: and only if α = p ± √ q for some p, q ∈ Q, q > 0, (b) D α ∈ S for any α ∈ R \ Q, (c) D α,r ∈ S for any α ∈ R \ Q, r > 1.
Suppose now that α = p n ± q n 2 for some p, q ∈ Z, q > 0, n ∈ N. We are looking for k i , l i ∈ Z, k 1 + αl 1 ≥ 0 such that (20) defines an automorphism of D * α . Then the following equations are satisfied: and |k 1 l 2 − k 2 l 1 | = 1.
Consider the so-called Pell's equation of the following form: It was shown by Lagrange that (22) has infinitely many integer solutions (recall that q is not a square of a natural number). Let x, y, where x, y > 0, denote the arbitrary natural solution of this equation. Put It is a direct consequence of (22) that x ± ny √ q > 0, hence k 1 + αl 1 > 0. An easy computation shows that each condition in (21) is satisfied by such chosen integers k i , l i , i = 1, 2. Moreover, so it follows from Theorem 1 in [23] that D / ∈ S. (b) It follows from Theorem 14 that any automorphism of D α is elementary algebraic. The automorphisms must also preserve the axis {0} × C * (when α > 0, the axis C × {0} is also preserved). From this piece of information one may conclude that any automorphism of domain D α is of the form where a, b ∈ C, |a||b| α = 1. Therefore, the functions u + , u − given by the formulas and u − (z 1 , z 2 ) = max{u + (z 1 , z 2 ), − log |z 2 |}, when α < 0, satisfy the criterion of Stehlé.

The Case when D ∩ C 2 * is Hyperbolic
For a pseudoconvex Reinhardt domain D in C n let I (D) denote the set of i = 1, . . . , n, for which the intersection (C i−1 ×{0}×C n−i )∩D is not hyperbolic (viewed as a domain in C n−1 ). Put Proof Take any sequence (ϕ n ) n ⊂ Aut(D). Since for any ϕ ∈ Aut(D) the restriction ϕ| D hyp is an automorphism of D hyp (see, e.g., [11], Theorem 8), we may assume that (ϕ n | D hyp ) n is convergent locally uniformly on D hyp . Applying Cauchy's formula we infer that (ϕ n ) n is convergent to some holomorphic function on D. Repeating the above argument for the sequence (ϕ −1 n ) n immediately gives the desired result.
The problem of a characterization of automorphism groups of bounded Reinhardt domains was studied in [18] (see also [7,8], and papers referenced there).
The results obtained in [18] and [12], together with remarks from [17], lead us to the description of pseudoconvex hyperbolic Reinhardt domains with t = 1 and non-compact automorphism group. For our future use we recall here the version formulated in [17].
Theorem 18 (See [12,18], and [17], Theorem 4) Let D be a hyperbolic, pseudoconvex Reinhardt domain with t (D) = 1. Then Aut(D) is non-compact if and only if D is algebraically equivalent to one of the following domains: (a) D×A(r, 1), where 0 ≤ r < 1. In this case, the group of automorphisms consists of the mappings of the form D (z 1 , z 2 ) → (a(z 1 ), b(z 2 )) ∈ D, where a ∈ Aut(D) and b ∈ Aut(A(r, 1)).
(b) {(z 1 , z 2 ) ∈ C 2 : |z 1 | < 1, 0 < |z 2 | < (1 − |z 1 | 2 ) p/2 }, p > 0. In this case, the group of automorphisms consists of the mappings of the form D (z 1 , z 2 ) → In this case the group of automorphisms consists of the mappings of the form D (z 1 , The following lemma will be also needed:  Lemma 19), any automorphism of the domain of one of the forms presented above preserves the axis C × {0}. This fact and Theorem 18 lead to the statement that the group of automorphisms of the domain D in all cases (a), (b), and (c) consists of the mappings of the form where |a| = |b| = 1. Therefore, Aut(D) is compact in all cases, which is a contradiction.
Let us pass to the remaining case t (D hyp ) = 0. An important role in our approach is played by the following result: where ψ : R → R is a concave function satisfying the property ψ(β + s) − ψ(s) = α + ks, s ∈ R for some α, β ∈ R, β = 0, k ∈ Z * .
Proof Using the inclusion Aut(D) |D hyp ⊂ Aut(D hyp ) (Lemma 19) and Theorem 21, we state that any automorphism of D must be algebraic. Moreover, at least one of the axes {0} × C * , C * × {0} must be contained in D (otherwise, D would be hyperbolic).
Suppose that both axes are contained in D. Then, since any automorphism of D maps the axes onto the axes (use Lemma 19), we see that the group Aut(D) consists of the mappings of the form for suitable a, b ∈ C * . We shall show that |a| = |b| = 1 in the case (28). Moreover, we shall prove that there is R > 0 such that for any automorphism satisfying (29), |a| = R and |b| = 1/R. This in particular means that Aut(D) is compact. Let us take any ϕ ∈ Aut(D) of the form (28). Assume the contrary, i.e., (log |a|, log |b|) = (0, 0). For n ∈ N put ϕ (n) = ϕ • · · · • ϕ ∈ Aut(D) and ϕ (−n) = (ϕ −1 ) (n) . Since ϕ (n) (z 1 , z 2 ) = (a n z 1 , b n z 2 ) ∈ D for any (z 1 , z 2 ) ∈ D, passing to the logarithmic image of the domain D easily shows that R(log |a|, log |b|) + log D ⊂ log D. This contradicts the hyperbolicity of D hyp .
This finishes the proof of the compactness of Aut(D) in this case. Suppose that only one axis is contained in D, e.g., {0} × C * ⊂ D. We use the idea applied by P. Pflug and W. Zwonek in [17].
Assume that the group Aut(D) is non-compact. Note that D ∩ (C × {0}) = ∅. As before, using the fact that automorphisms preserve the axis (Lemma 19), we see that Aut(D) consists of the mappings of the form = a,b,k, : (z 1 , z 2 ) → az 1 z k 2 , bz 2 for some a, b ∈ C * , = ±1 and k ∈ Z.
( †) We shall show that there exists an automorphism a,b,k, of the domain D with |b| = 1 and = 1 (then also k = 0).
First observe that there is an automorphism of the domain D of the form (30) with k = 0. To show this, repeat the argument from the first part of the proof to observe that |a| = |b| = 1 for any automorphism a,b,0,1 . Moreover, there exists R > 0 such that |a| = 1, |b| = R for any automorphism a,b,0,−1 . So if there were no automorphisms of the form (30) with k = 0, then the group Aut(D) would be compact.
If there is an automorphism of D of the form (30) with = 1 and k = 0, we are done. Actually, (n) (z 1 , z 2 ) = (a n b kn(n−1)/2 z 1 z nk 2 , b n z 2 ), and using the hyperbolicity of D hyp once again, we find that |b| = 1.
So we have shown ( †). The properties of pseudoconvex Reinhardt domains imply that for any s ∈ R there is (exactly one) ψ(s) ∈ R such that (ψ(s), s) ∈ ∂ log D. Moreover, it is an immediate consequence of the inclusion {0} × C * ⊂ D and the convexity of log D that the function ψ(·) is concave. Put v(z 1 , z 2 ) = log |z 1 | − ψ(log |z 2 |).
A direct calculation (approximate and compute the Levi form) shows that u is plurisubharmonic. Moreover, u is an exhaustion function for D. In view of (32) u satisfies the criterion of Stehlé.
Using ( †) and the property (31), once again we find that ψ(log |b| + s) − ψ(s) = log |a| + ks. From this we obtain the desired properties of the function ψ .
Conversely, having a domain D ⊂ C × C * satisfying (27) one may easily see that the mapping given by the formula (30) with b = e β , a = e α , = 1 is an automorphism of the domain D. The investigation of (n) immediately proves the non-compactness of Aut(D).

C × C *
In 1977 a negative answer to the Serre problem was given by H. Skoda, who proved that C 2 / ∈ S. This construction was improved in [2] by J.P. Demailly, who proved that polynomial automorphisms of C 2 may serve as the transition function. Later, in [3], J.P. Demailly constructed a counterexample to the Serre problem with a plane or a disc as a base.
Let us recall here this construction. The base is a domain containing 3D. Put where M(V , ω, r) = max ω×(rD) 2 V and the constant C does not depend on r. The key role was played by the logarithmic convexity of the functions (ρ, r) → M(V , ρD, r), V ∈ PSH( × C 2 ).
Direct computation allows us to obtain the logarithmic convexity of the function (ρ, r) → max ρD×rD×A(r)Ṽ for anyṼ ∈ PSH( × C × C * ). Therefore, considering M instead of M, whereM(V , ω, r) = max ω×rD×A(r) V , and repeating the reasoning from Demailly's paper, we may replace M byM in the inequality (33). This, together with the above-mentioned logarithmic convexity ofM, immediately shows that C × C * does not belong to S.

Proofs of the Main Theorems
Proof of Theorem 1 The result follows from Theorems 12, 15, 20, and 22.

Proof of Theorem 3
It is clear that if log D contains an affine line, then the group Aut(D) is non-compact.
In the case when log D contains no affine lines (i.e., D hyp hyperbolic) the result follows immediately from Theorems 20 and 22.