Modularizing

As we grow and support the suite of Cloud services, this repo could get very big, leading to:

unnecessary download size for consumers that just care about 1/n services,
maintainability headaches.

I propose we split out into multiple repos, each representing a service (eg, gcloud-datastore, gcloud-storage, etc.). If we have common functionality to share between services, we can expose that as a module as well.

We discussed this issue a while ago and ended up with bundling all clients into a single module.

unnecessary download size for consumers that just care about 1/n services

We don't think it'll be an issue since clients are thin. A few more kilobytes won't hurt. We have plans to depend on Protobuf.js, it may be a game changer though.

If we have common functionality to share between services, we can expose that as a module as well.

That will make the release process unnecessarily complicated in our scale. Maintainers doesn't want to waste my time trying to version small tiny packages, and keep track of who should depend what and which version.

I'm just here to help, so I won't fight too hard (:punch:!). However, the way you're suggesting goes against the Node/npm philosophy of small (the tinier the better!), maintainable, reusable modules. This is the ecosystem that is already in place, and one developers have come to expect when developing and consuming packages. Believe it or not, people do raise concern when a package grows to a larger kb size, but the big miss with our approach, that I don't think should be overlooked, is maintainability of this repo. When it grows to a size that all services are supported, this repo ends up with overlapping PRs and issues. It has been proven more effective to split out the "modules" within a large repo, allowing for maintainers with specialities in the individual services to continue offering support. Again, I'll follow the leader, but wouldn't mind hearing more thoughts considering these points :)

@jgeewax

@jgeewax @callmehiphop is this worth re-consideration? I've been splitting out pieces here and there (auth, resumable upload, gce-images, etc), and that's simply because if we kept all that stuff in this repo, nobody would ever want to contribute (or in the future for others, maintain). Now that we have quite a few services, with at least 4 more planned soon, maybe we should quit bloating up gcloud with unrelated code and modularize out?

It would be much easier for someone to get started if they could just npm install --save gcloud-resource-manager. Their use case could very possibly only need 25 lines of code from us, yet with our current configuration, we would force them to download thousands, plus modules their service doesn't need.

The gcloud package can live on as a wrapper around all of the modules. The one big show-stopper I can think of is, how would we integrate separate modules into our gcloud docs? We would have to be diligent about updating gcloud's package.json with any bumps from our modules.

Are the cons worth the pros? Excited to hear your thoughts!

I'm all for modularizing the library, we've discussed this before and as it stands today, we're probably breaking some node best practices by bundling several modules together.

I think the obvious pros are:

We become more aligned with node best practices
We decrease the overall download size of the library
We would no longer require the user to install unnecessary and cumbersome dependencies (sse4_crc32, grpc, etc.) which could provide a better user experience when installing gcloud.

As mentioned previously, a big con to this approach would be the added complexity to our release process. However, I believe there have been several discussions around making the upcoming bigtable api it's own module due to it's dependency on grpc, so the added complexity will probably end up being unavoidable anyways.

We decrease the overall download size of the library

If we use gcloud as a wrapper, it could end up being more. npm 3.x dedupes by default, but in older versions, every module that requires through for example, will end up with the user getting that many more copies of through... until the user manually runs npm dedupe... if they even care.

However, I believe there have been several discussions around making the upcoming bigtable api it's own module due to it's dependency on grpc, so the added complexity will probably end up being unavoidable anyways.

That's a good point.

(Relevant: googleapis/google-cloud-python#1094 (comment)): I was thinking the gcloud-common docs site could be the official "gcloud libraries docs" with different paths to show only one language's docs, for example:

http://googlecloudplatform.github.io/gcloud-common/#/node === Node's API docs

That would pull JSON files right from our repo, e.g.: gcloud-node/gh-pages/json/docs/(latestVersion).json

We could format our external modules to go through the same docs -> json transform. We would iterate over the services gcloud supports, and show links for them in the nav bar. When an item is selected, it loads that version's json from the service's repo, e.g: gcloud-storage/gh-pages/json/docs/(version tagged from gcloud's package.json).json

But a user who just wants to use our gcloud-storage module needs docs as well, so our same endpoint should be able to work as "The official documentation of the storage module":

http://googlecloudplatform.github.io/gcloud-common/#/node/storage

This might be crazy.

I'm not a huge fan of having gcloud-common in the url.

I think that might be our only option... that I can think of anyway. Each module would just redirect to the common URL. Copying the template over to every modules gh-pages would be pretty annoying.

I'm definitely not saying we should copy over the template to every gh-pages, rather we could use hooks or submodules to inject the json into gcloud-node and then in the individual repositories just link to the appropriate api.. /gcloud-node/docs/storage/v0.x.0

Also, something like this would have an impact on a common template for all the other apis since we'd need to offer version switching at the individual module level - which as of today no one else does.

I'm trying to look out for the other gclouds. If we all share a template, it would be great to not have to individually host it. I haven't received feedback on that idea however.

Please outline a plan that you see working and let's discuss that one :)