| commit | b3b96b2fceb93cc270ece419c3d0dc9df4a71012 | [log] [tgz] |
|---|---|---|
| author | Emmanuel T Odeke <emmanuel@orijtech.com> | Wed Apr 15 22:29:20 2020 -0700 |
| committer | Chris Cotter <cjcotter@google.com> | Fri Apr 17 02:54:02 2020 +0000 |
| tree | 9e6a4fd7226b02ffde13a932b459cf73f7801851 | |
| parent | 93838443a233d369c08fb12115cf05e17785f070 [diff] |
storage: properly read files with Content-Encoding gzip
Dictated by Cloud Storage's "decompressive transcoding" advisory:
https://cloud.google.com/storage/docs/transcoding
If a file is stored with
"Content-Encoding": "gzip"
that file is stored on Cloud Storage in the compressed form but when
downloaded by clients it will be decompressed on the fly and
served in the decompressed form.
That also means that range requests for such files are ignored by Cloud Storage.
This change removes the prior assumption that EVERY range
request would result in a 206 Partial Content response,
yet in this case a whole file is always served regardless
of the range for files with "Content-Encoding": "gzip".
Instead such files are served back with a 200 status code.
Due to the consequence of files being served wholly by Cloud Storage
irregardless of the range request, we manually consume "seen" bytes from
retried bodies so that we only send back the caller the remaining bytes.
This change adds both integration and behavioral/mocking tests
to ensure that the behavior is always the same when dealing with
such files.
Also manually verified by testing out a repro that was created
for the issue and checksumming the response. The reproducer isn't
included in the tests because it requires the server to panic,
causing an HTTP/2 stream error, then perfom the retry. However,
the results of serving the retried file, after MD5 checksumming a file
containing 1 million lines of "I am a line\n":
GotMD5: 90d6ab70c7e44d6b3ccdec8658b56f69
WantMD5: 90d6ab70c7e44d6b3ccdec8658b56f69
Fixes #1800
Change-Id: I858e547e5faa5d2caa70b18b64f4d03dcaac8270
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/54791
Reviewed-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Chris Cotter <cjcotter@google.com>
Reviewed-by: Tyler Bui-Palsulich <tbp@google.com>
Reviewed-by: Frank Natividad <franknatividad@google.com>
Go packages for Google Cloud Platform services.
import "cloud.google.com/go"
To install the packages on your system, do not clone the repo. Instead:
Change to your project directory:
cd /my/cloud/project
Get the package you want to use. Some products have their own module, so it's best to go get the package(s) you want to use:
$ go get cloud.google.com/go/firestore # Replace with the package you want to use.
NOTE: Some of these packages are under development, and may occasionally make backwards-incompatible changes.
NOTE: Github repo is a mirror of https://code.googlesource.com/gocloud.
Alpha status: the API is still being actively developed. As a result, it might change in backward-incompatible ways and is not recommended for production use.
Beta status: the API is largely complete, but still has outstanding features and bugs to be addressed. There may be minor backwards-incompatible changes where necessary.
Stable status: the API is mature and ready for production use. We will continue addressing bugs and feature requests.
Documentation and examples are available at pkg.go.dev/cloud.google.com/go
We support the two most recent major versions of Go. If Google App Engine uses an older version, we support that as well.
By default, each API will use Google Application Default Credentials for authorization credentials used in calling the API endpoints. This will allow your application to run in many environments without requiring explicit configuration.
client, err := storage.NewClient(ctx)
To authorize using a JSON key file, pass option.WithCredentialsFile to the NewClient function of the desired package. For example:
client, err := storage.NewClient(ctx, option.WithCredentialsFile("path/to/keyfile.json"))
You can exert more control over authorization by using the golang.org/x/oauth2 package to create an oauth2.TokenSource. Then pass option.WithTokenSource to the NewClient function: snip:# (auth-ts)
tokenSource := ...
client, err := storage.NewClient(ctx, option.WithTokenSource(tokenSource))
Contributions are welcome. Please, see the CONTRIBUTING document for details. We‘re using Gerrit for our code reviews. Please don’t open pull requests against this repo, new pull requests will be automatically closed.
Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms. See Contributor Code of Conduct for more information.