support/bundler: fix deadlock when using time based flush

bundler.Bundler can run into a deadlock under the following
configuration:

- handler limit of 1
- high size and count thresholds such that the only reason a bundle is
flushed is because of the delay timer expiring

Under these conditions, with high concurrency, a deadlock can happen
because the background handler goroutine can exit while there is a
pending queue, which will never get flushed.

Specifically, this sequence of events:

- handler goroutine calls postHandle and receives a nil for the next
bundle, signaling it has no more work to do.

- some other goroutine calls AddWait() -> enqueueCurBundle, which sees
that b.handlerLimit cannot be exceeded and proceeds to queue the item

- handler goroutine decrements b.handlerLimit and exits

The fix is to move the decrement of b.handlerLimit into postHandle such
that the decision to exit and allowing a new handler goroutine to be
spawned cannot be re-ordered.

Fixes: googleapis/google-api-go-client#475

Change-Id: I84421e53a4dc24eec36ceb147933aab57a40aec8
Reviewed-on: https://code-review.googlesource.com/c/google-api-go-client/+/55950
Reviewed-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Cody Oss <codyoss@google.com>
2 files changed
tree: c51cd2205c6fd6c148f032a4b7c9783dd68c335c
  1. .gitignore
  2. .hgtags
  3. AUTHORS
  4. CHANGES.md
  5. CONTRIBUTING.md
  6. CONTRIBUTORS
  7. GettingStarted.md
  8. LICENSE
  9. NOTES
  10. README.md
  11. RELEASING.md
  12. TODO
  13. abusiveexperiencereport/
  14. acceleratedmobilepageurl/
  15. accessapproval/
  16. accesscontextmanager/
  17. adexchangebuyer/
  18. adexchangebuyer2/
  19. adexchangeseller/
  20. adexperiencereport/
  21. admin/
  22. admob/
  23. adsense/
  24. adsensehost/
  25. alertcenter/
  26. analytics/
  27. analyticsreporting/
  28. androiddeviceprovisioning/
  29. androidenterprise/
  30. androidmanagement/
  31. androidpublisher/
  32. api-list.json
  33. apigee/
  34. appengine/
  35. appsactivity/
  36. appstate/
  37. bigquery/
  38. bigqueryconnection/
  39. bigquerydatatransfer/
  40. bigqueryreservation/
  41. bigtableadmin/
  42. billingbudgets/
  43. binaryauthorization/
  44. blogger/
  45. books/
  46. calendar/
  47. chat/
  48. civicinfo/
  49. classroom/
  50. cloudasset/
  51. cloudbilling/
  52. cloudbuild/
  53. cloudcommerceprocurement/
  54. clouddebugger/
  55. clouderrorreporting/
  56. cloudfunctions/
  57. cloudidentity/
  58. cloudiot/
  59. cloudkms/
  60. cloudprivatecatalog/
  61. cloudprivatecatalogproducer/
  62. cloudprofiler/
  63. cloudresourcemanager/
  64. cloudscheduler/
  65. cloudsearch/
  66. cloudshell/
  67. cloudtasks/
  68. cloudtrace/
  69. commentanalyzer/
  70. composer/
  71. compute/
  72. consumersurveys/
  73. container/
  74. containeranalysis/
  75. content/
  76. customsearch/
  77. datacatalog/
  78. dataflow/
  79. datafusion/
  80. dataproc/
  81. datastore/
  82. deploymentmanager/
  83. dfareporting/
  84. dialogflow/
  85. digitalassetlinks/
  86. discovery/
  87. displayvideo/
  88. dlp/
  89. dns/
  90. doc.go
  91. docs/
  92. domainsrdap/
  93. doubleclickbidmanager/
  94. doubleclicksearch/
  95. drive/
  96. driveactivity/
  97. examples/
  98. factchecktools/
  99. fcm/
  100. file/
  101. firebase/
  102. firebasedynamiclinks/
  103. firebasehosting/
  104. firebaseml/
  105. firebaseremoteconfig/
  106. firebaserules/
  107. firestore/
  108. fitness/
  109. fusiontables/
  110. games/
  111. gamesconfiguration/
  112. gameservices/
  113. gamesmanagement/
  114. genomics/
  115. gmail/
  116. go.mod
  117. go.sum
  118. google-api-go-generator/
  119. googleapi/
  120. groupsmigration/
  121. groupssettings/
  122. healthcare/
  123. homegraph/
  124. iam/
  125. iamcredentials/
  126. iap/
  127. identitytoolkit/
  128. idtoken/
  129. indexing/
  130. integration-tests/
  131. internal/
  132. iterator/
  133. jobs/
  134. kgsearch/
  135. language/
  136. lib/
  137. libraryagent/
  138. license_test.go
  139. licensing/
  140. lifesciences/
  141. logging/
  142. managedidentities/
  143. manufacturers/
  144. memcache/
  145. mirror/
  146. ml/
  147. monitoring/
  148. networkmanagement/
  149. oauth2/
  150. option/
  151. osconfig/
  152. oslogin/
  153. pagespeedonline/
  154. partners/
  155. people/
  156. playcustomapp/
  157. playmoviespartner/
  158. plus/
  159. plusdomains/
  160. policytroubleshooter/
  161. poly/
  162. prod_tt_sasportal/
  163. proximitybeacon/
  164. pubsub/
  165. qpxexpress/
  166. recommender/
  167. redis/
  168. remotebuildexecution/
  169. replicapool/
  170. replicapoolupdater/
  171. reseller/
  172. run/
  173. runtimeconfig/
  174. safebrowsing/
  175. sasportal/
  176. script/
  177. searchconsole/
  178. secretmanager/
  179. securitycenter/
  180. servicebroker/
  181. serviceconsumermanagement/
  182. servicecontrol/
  183. servicedirectory/
  184. servicemanagement/
  185. servicenetworking/
  186. serviceusage/
  187. serviceuser/
  188. sheets/
  189. siteverification/
  190. slides/
  191. sourcerepo/
  192. spanner/
  193. spectrum/
  194. speech/
  195. sql/
  196. sqladmin/
  197. storage/
  198. storagetransfer/
  199. streetviewpublish/
  200. support/
  201. surveys/
  202. tagmanager/
  203. tasks/
  204. testing/
  205. texttospeech/
  206. toolresults/
  207. tools.go
  208. tpu/
  209. tracing/
  210. translate/
  211. transport/
  212. urlshortener/
  213. vault/
  214. verifiedaccess/
  215. videointelligence/
  216. vision/
  217. webfonts/
  218. webmasters/
  219. websecurityscanner/
  220. youtube/
  221. youtubeanalytics/
  222. youtubereporting/
README.md

Google APIs Client Library for Go

Getting Started

$ go get google.golang.org/api/tasks/v1
$ go get google.golang.org/api/moderator/v1
$ go get google.golang.org/api/urlshortener/v1
... etc ...

and using:

package main

import (
	"net/http"

	"google.golang.org/api/urlshortener/v1"
)

func main() {
	svc, err := urlshortener.New(http.DefaultClient)
	// ...
}

Status

GoDoc

These are auto-generated Go libraries from the Google Discovery Service's JSON description files of the available “new style” Google APIs.

Due to the auto-generated nature of this collection of libraries, complete APIs or specific versions can appear or go away without notice. As a result, you should always locally vendor any API(s) that your code relies upon.

These client libraries are officially supported by Google. However, the libraries are considered complete and are in maintenance mode. This means that we will address critical bugs and security issues but will not add any new features.

If you're working with Google Cloud Platform APIs such as Datastore or Pub/Sub, consider using the Cloud Client Libraries for Go instead. These are the new and idiomatic Go libraries targeted specifically at Google Cloud Platform Services.

The generator itself and the code it produces are beta. Some APIs are alpha/beta, and indicated as such in the import path (e.g., “google.golang.org/api/someapi/v1alpha”).

Application Default Credentials Example

Application Default Credentials provide a simplified way to obtain credentials for authenticating with Google APIs.

The Application Default Credentials authenticate as the application itself, which make them great for working with Google Cloud APIs like Storage or Datastore. They are the recommended form of authentication when building applications that run on Google Compute Engine or Google App Engine.

Default credentials are provided by the golang.org/x/oauth2/google package. To use them, add the following import:

import "golang.org/x/oauth2/google"

Some credentials types require you to specify scopes, and service entry points may not inject them. If you encounter this situation you may need to specify scopes as follows:

import (
        "context"
        "golang.org/x/oauth2/google"
        "google.golang.org/api/compute/v1"
)

func main() {
        // Use oauth2.NoContext if there isn't a good context to pass in.
        ctx := context.Background()

        client, err := google.DefaultClient(ctx, compute.ComputeScope)
        if err != nil {
                //...
        }
        computeService, err := compute.New(client)
        if err != nil {
                //...
        }
}

If you need a oauth2.TokenSource, use the DefaultTokenSource function:

ts, err := google.DefaultTokenSource(ctx, scope1, scope2, ...)
if err != nil {
        //...
}
client := oauth2.NewClient(ctx, ts)

See also: golang.org/x/oauth2/google package documentation.