)]}' { "commit": "9e18ddb094c17cfb790e6544a4fea6716b907784", "tree": "30d2f1fb7cd37bef9a7e52c08d3bde407febe792", "parents": [ "f013ab08d24b21e75e8e6aa7125de4336b705b1f" ], "author": { "name": "Jean de Klerk", "email": "deklerk@google.com", "time": "Fri Jun 07 17:02:15 2019 -0600" }, "committer": { "name": "Jean de Klerk", "email": "deklerk@google.com", "time": "Thu Oct 10 10:03:16 2019 -0600" }, "message": "pubsub: add ordered keys\n\nPub/Sub ordered keys requires:\n\n- Publish with a non-empty key gets added to a per-key queue, as implemented\nby Bundler.HandlerLimit\u003d1. This setting causes the bundler to only have one\noutstanding bundle being handled at a time; further bundles form a queue.\n- Receive with a message that has a non-empty key gets added to a per-key\nqueue, as implemented by a slice. When a worker operates on a key, it\nprocesses all items in the key\u0027s slice until the slice is empty, at which point\nit deletes the key to release the key\u0027s resources, and the worker returns to\nthe worker pool.\n\nFunctionally, for users, this should involve:\n\n- Slightly slower processing speeds due to additional data structures\nalong the Publish and Receive paths. (we should do some performance\ntesting to vet this out)\n- PublishSettings.NumGoroutines and ReceiveSettings.NumGoroutines should\nboth result in double the number of goroutines, since an additional data\nstructure (publish/receive scheduler) uses that value. Documentation has\nbeen adjusted accordingly to describe it more as a scaling factor than\nan exact number.\n- Default ReceiveSettings.NumGoroutines is increased from 1 to 10. It\u0027s\ngenerally a better experience to have multiple workers than one. And,\nsome tests - like TestStreamingPullFlowControl - implicitly require it.\n- Since ordered keys require only a single outstanding RPC at once, it is\npossible to send ordered key messages to Topic.Publish (and subsequently to\nPublishScheduler.Add) faster than the bundler can publish them to the\nPub/Sub service, resulting in a backed up queue of Pub/Sub bundles. Each\nitem in the bundler queue is a goroutine, which means users may see thousands\nof goroutines in debugging they might do.\n\nTODO: add Resume method.\n\nChange-Id: Ib710944d557970290cf12321d1fdbd9b1699231d\n", "tree_diff": [ { "type": "modify", "old_id": "03bad6e3c623256c2a066e35c93cb81586f13df5", "old_mode": 33188, "old_path": "pubsub/integration_test.go", "new_id": "bc7f28f3acb2138d9ae5bb4bef164ac794352e76", "new_mode": 33188, "new_path": "pubsub/integration_test.go" }, { "type": "add", "old_id": "0000000000000000000000000000000000000000", "old_mode": 0, "old_path": "/dev/null", "new_id": "df4950d1d45608c1fce933b07120d1ccd45a2b8c", "new_mode": 33188, "new_path": "pubsub/internal/scheduler/publish_scheduler.go" }, { "type": "add", "old_id": "0000000000000000000000000000000000000000", "old_mode": 0, "old_path": "/dev/null", "new_id": "421096aedc8e444101162e524b950f5834cae7ed", "new_mode": 33188, "new_path": "pubsub/internal/scheduler/publish_scheduler_benchmark_test.go" }, { "type": "add", "old_id": "0000000000000000000000000000000000000000", "old_mode": 0, "old_path": "/dev/null", "new_id": "0af0837aae90157617f424a6d2b2b09d5e51e786", "new_mode": 33188, "new_path": "pubsub/internal/scheduler/publish_scheduler_test.go" }, { "type": "add", "old_id": "0000000000000000000000000000000000000000", "old_mode": 0, "old_path": "/dev/null", "new_id": "3a8d9b83db515570af4b17c96ead04a0895ff5b0", "new_mode": 33188, "new_path": "pubsub/internal/scheduler/receive_scheduler.go" }, { "type": "add", "old_id": "0000000000000000000000000000000000000000", "old_mode": 0, "old_path": "/dev/null", "new_id": "674f9e6ea74586f73c78b60e5ab3de2a25102706", "new_mode": 33188, "new_path": "pubsub/internal/scheduler/receive_scheduler_benchmark_test.go" }, { "type": "add", "old_id": "0000000000000000000000000000000000000000", "old_mode": 0, "old_path": "/dev/null", "new_id": "de03300400d844fe007efbf73f802dde4b7ceba2", "new_mode": 33188, "new_path": "pubsub/internal/scheduler/receive_scheduler_test.go" }, { "type": "add", "old_id": "0000000000000000000000000000000000000000", "old_mode": 0, "old_path": "/dev/null", "new_id": "66ab4fa6bacadada57f04c0289ea81cc7f567f91", "new_mode": 33188, "new_path": "pubsub/internal/testutil/verifier.go" }, { "type": "add", "old_id": "0000000000000000000000000000000000000000", "old_mode": 0, "old_path": "/dev/null", "new_id": "7a381e5c2282253fe546c01e21361be9fd07e191", "new_mode": 33188, "new_path": "pubsub/internal/testutil/verifier_test.go" }, { "type": "modify", "old_id": "c4b16e95f164695990796118348815a1f171dcc0", "old_mode": 33188, "old_path": "pubsub/message.go", "new_id": "77c51c8c6dcf409969399025cb1e3bfab984a9db", "new_mode": 33188, "new_path": "pubsub/message.go" }, { "type": "modify", "old_id": "52da17407cf684fe1b29c154a47a5352d2c8f8ae", "old_mode": 33188, "old_path": "pubsub/mock_test.go", "new_id": "c72ee6e764a3b99c0a7550f29d22894767be5d3b", "new_mode": 33188, "new_path": "pubsub/mock_test.go" }, { "type": "modify", "old_id": "5364387cee45b7adf5b4606de2405a2683d759a5", "old_mode": 33188, "old_path": "pubsub/streaming_pull_test.go", "new_id": "5c49d6ec6694408ecd9c1972df7073312e907ef5", "new_mode": 33188, "new_path": "pubsub/streaming_pull_test.go" }, { "type": "modify", "old_id": "bfd9dfb9279b4f1174ec769b94c7c197e53339a4", "old_mode": 33188, "old_path": "pubsub/subscription.go", "new_id": "68fa408f7b3fec12825903c10d435a89e894fa4d", "new_mode": 33188, "new_path": "pubsub/subscription.go" }, { "type": "add", "old_id": "0000000000000000000000000000000000000000", "old_mode": 0, "old_path": "/dev/null", "new_id": "9b28a9c505cdcaef2ccecb286ae7821be760df31", "new_mode": 33188, "new_path": "pubsub/testdata/README.md" }, { "type": "add", "old_id": "0000000000000000000000000000000000000000", "old_mode": 0, "old_path": "/dev/null", "new_id": "bdfe3639f8bcf96bd84b0e839cb6669aa1aef129", "new_mode": 33188, "new_path": "pubsub/testdata/publish.csv" }, { "type": "modify", "old_id": "e03e227b948915a08f40d0fd7419e38447e2e8fc", "old_mode": 33188, "old_path": "pubsub/topic.go", "new_id": "a78434e8c9da7215dcd172316a3b3820548198c7", "new_mode": 33188, "new_path": "pubsub/topic.go" } ] }