grpc: add WithWaitForHandshake

This change causes grpc-go to wait for the server to respond with a connection preface ("i'm properly configured and ready to communicate") rather than the default behavior of attempting to send data on a connection that has only been Dial'ed.

This may be related to the recent Pub/Sub issues. For example, perhaps the clients are trying to send large amounts of messages on servers that are not ready, causing the GFE to push back.

This is a fairly outlandish theory, but luckily this change has no negative consequence, and if nothing else it makes the code easier to reason about (rather than having to consider what's happening to data being sent on a non-ready connection).

This will become a default setting in a later version of grpc-go.

Change-Id: Ic6570a638f6f0b20d424b6c5c2fe31703690a798
Reviewed-on: https://code-review.googlesource.com/c/34410
Reviewed-by: Jonathan Amsterdam <jba@google.com>
diff --git a/transport/grpc/dial.go b/transport/grpc/dial.go
index c79b835..187b932 100644
--- a/transport/grpc/dial.go
+++ b/transport/grpc/dial.go
@@ -59,7 +59,9 @@
 	if o.GRPCConn != nil {
 		return o.GRPCConn, nil
 	}
-	var grpcOpts []grpc.DialOption
+	grpcOpts := []grpc.DialOption{
+		grpc.WithWaitForHandshake(),
+	}
 	if insecure {
 		grpcOpts = []grpc.DialOption{grpc.WithInsecure()}
 	} else if !o.NoAuth {