| commit | 000766f8a65eb391dc85f5181c7b2148f7a08e22 | [log] [tgz] |
|---|---|---|
| author | Eddie Scholtz <escholtz@google.com> | Wed Nov 06 10:09:05 2019 -0700 |
| committer | Jean de Klerk <deklerk@google.com> | Thu Nov 21 16:04:31 2019 +0000 |
| tree | b97cb575e7cc4b8df4364634f98e844d5542d8bb | |
| parent | 006ab4afe25e899e4ec545ce1cbfdaed675d15d8 [diff] |
bundler: limit the total number of goroutines to HandlerLimit The bundler creates a goroutine per bundle. Only HandlerLimit can execute in parallel so when write rates are high and handling is slow, there can be a buildup of goroutines that are competing to determine if it is their turn to execute. This behavior is exhibited by TestConcurrentHandlersMax which occasionally fails to terminate and generates 250,000+ goroutines within 2 seconds on a 6-core 3.7ghz linux desktop. In this change, we limit the total number of goroutines to HandlerLimit. We change the bundle type to be a linked list where items are added to the tail and handled from the head. Once the user-specified handler returns within the goroutine, we check if another bundle is ready for handling. If a bundle is ready, the existing goroutine handles it. Otherwise, the goroutine ends. This allows the goroutine count to grow and shrink as necessary. Since we avoid sending bundles to goroutines prematurely, we increase our opportunity to fill bundles to their maximum item count. This change allows bundler tests to execute and pass when GOMAXPROCS=1. Previously they would fail to terminate. Fixes #367 Fixes #417 Change-Id: I73a236ffe49087aedebbbbb6bbc94e0a041e93c1 Reviewed-on: https://code-review.googlesource.com/c/google-api-go-client/+/47991 Reviewed-by: kokoro <noreply+kokoro@google.com> Reviewed-by: Jean de Klerk <deklerk@google.com>
$ go get google.golang.org/api/tasks/v1 $ go get google.golang.org/api/moderator/v1 $ go get google.golang.org/api/urlshortener/v1 ... etc ...
and using:
package main import ( "net/http" "google.golang.org/api/urlshortener/v1" ) func main() { svc, err := urlshortener.New(http.DefaultClient) // ... }
These are auto-generated Go libraries from the Google Discovery Service's JSON description files of the available “new style” Google APIs.
Due to the auto-generated nature of this collection of libraries, complete APIs or specific versions can appear or go away without notice. As a result, you should always locally vendor any API(s) that your code relies upon.
These client libraries are officially supported by Google. However, the libraries are considered complete and are in maintenance mode. This means that we will address critical bugs and security issues but will not add any new features.
If you're working with Google Cloud Platform APIs such as Datastore or Pub/Sub, consider using the Cloud Client Libraries for Go instead. These are the new and idiomatic Go libraries targeted specifically at Google Cloud Platform Services.
The generator itself and the code it produces are beta. Some APIs are alpha/beta, and indicated as such in the import path (e.g., “google.golang.org/api/someapi/v1alpha”).
Application Default Credentials provide a simplified way to obtain credentials for authenticating with Google APIs.
The Application Default Credentials authenticate as the application itself, which make them great for working with Google Cloud APIs like Storage or Datastore. They are the recommended form of authentication when building applications that run on Google Compute Engine or Google App Engine.
Default credentials are provided by the golang.org/x/oauth2/google package. To use them, add the following import:
import "golang.org/x/oauth2/google"
Some credentials types require you to specify scopes, and service entry points may not inject them. If you encounter this situation you may need to specify scopes as follows:
import ( "context" "golang.org/x/oauth2/google" "google.golang.org/api/compute/v1" ) func main() { // Use oauth2.NoContext if there isn't a good context to pass in. ctx := context.Background() client, err := google.DefaultClient(ctx, compute.ComputeScope) if err != nil { //... } computeService, err := compute.New(client) if err != nil { //... } }
If you need a oauth2.TokenSource, use the DefaultTokenSource function:
ts, err := google.DefaultTokenSource(ctx, scope1, scope2, ...) if err != nil { //... } client := oauth2.NewClient(ctx, ts)
See also: golang.org/x/oauth2/google package documentation.