Skip to content

Makes checking fstat for file optional. - #914

Merged
dhermes merged 1 commit into
googleapis:masterfrom
dhermes:fix-900
Jun 18, 2015
Merged

dhermes merged 1 commit into
googleapis:masterfrom
dhermes:fix-900

Conversation

@dhermes

@dhermes dhermes commented Jun 6, 2015

Copy link
Copy Markdown
Contributor

In storage, file upload will now fail if no size is passed in and if the file object does not have a fileno
which can be stat'ed.

Fixes #900.

In storage, file upload will now fail if no size is
passed in and if the file object does not have a fileno
which can be stat'ed.

Fixes googleapis#900.
@ghost ghost added api: storage Issues related to the Cloud Storage API. cla: yes This human has signed the Contributor License Agreement. labels Jun 6, 2015

ghost commented Jun 6, 2015

Copy link
Copy Markdown
Author

@jonparrott This doesn't quite go all the way to fix what you wanted. It still requires size if the file object is not a file.

ghost commented Jun 6, 2015

Copy link
Copy Markdown

@Capstan : There's no way for GCS to just "not care about the size, we'll tell you when we're done piping data into it", right?

ghost commented Jun 6, 2015

Copy link
Copy Markdown
Author

It's an HTTP header thing.

ghost commented Jun 6, 2015

Copy link
Copy Markdown

Code LGTM.

The comment to @Capstan is in the hopes that we can send chunks until we're done sending chunks, and GCS counts the total bytes then... done and done. (That way, a stream that doesn't have a size up front, would be uploadable... right?)

ghost commented Jun 7, 2015

Copy link
Copy Markdown

re: @jgeewax

On the contrary, there are two published ways to send without knowing a final size.

The first works with single-POST / single-PUT (simple and multipart) and just uses Transfer-Encoding: chunked. Logically, we should even be able to handle trailing headers with digests (e.g., Content-MD5). We don't support the Digest header yet. Multipart requires doing the transfer encoding twice since you know neither the whole body nor the second body of the multipart's size.

The second works with resumable uploads. You omit the X-Upload-Content-Length header, and then use Content-Range: x-y/* (the star at the end to show you don't know the length) on the PUTs to the resumable session.

ghost commented Jun 7, 2015

Copy link
Copy Markdown

OK so then the question becomes: if I can send chunks until I'm done sending chunks, what's stopping us from uploading an arbitrary stream?

(I think this conversation is deviating from the topic of this PR though, so maybe best to discuss in a separate issue? The original maybe?)

ghost commented Jun 7, 2015

Copy link
Copy Markdown

This at least helps from a user perspective (way better error message) so this LGTM. However, I agree with @jgeewax - we should just be able to upload an arbitrary stream.

ghost commented Jun 9, 2015

Copy link
Copy Markdown

LGTM

ghost commented Jun 18, 2015

Copy link
Copy Markdown
Author

Opened #931 to capture @Capstan feedback and try to loop support for it into the library.

@ghost
ghost merged commit b0aef16 into googleapis:master Jun 18, 2015
@ghost
ghost deleted the fix-900 branch June 18, 2015 20:11
@ghost ghost mentioned this pull request Jul 10, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: storage Issues related to the Cloud Storage API. cla: yes This human has signed the Contributor License Agreement.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants