Local Installation

If you are just getting started with code generation for protobuf-based APIs, or if you do not have a robust Python environment already available, it is probably easier to get started using Docker: Docker Image

However, this tool offers first-class support for local execution using protoc. It is still reasonably easy, but initial setup will take a bit longer.

Note

If you are interested in contributing, setup according to these steps is recommended.

Installing

protoc

This tool is implemented as a plugin to the protocol buffers compiler, so in order to use it, you will need to have the protoc command available.

The release page on GitHub contains the download you need.

Note

You may notice both packages that designate languages (e.g. protobuf-python-X.Y.Z.tar.gz) as well as packages that designate architectures (e.g. protoc-X.Y.Z-linux-x86_64.zip). You want the one that designates an architecture; your goal here is to have a CLI command.

It is likely preferable to install protoc somewhere on your shell’s path, but this is not a strict requirement (as you will be invoking it directly). protoc is also quirky about how it handles well-known protos; you probably also want to copy them into /usr/local/include

To ensure it is installed propertly:

$ protoc --version
libprotoc 3.6.0

pandoc

This generator relies on pandoc to convert from Markdown (the lingua franca for documentation in protocol buffers) into ReStructured Text (the lingua franca for documentation in Python).

Install this using an appropriate mechanism for your operating system. Multiple installation paths are documented on the pandoc installation page.

API Generator for Python

This package is provided as a standard Python library, and can be installed the usual ways. It fundamentally provides a CLI command, protoc-gen-python_gapic, (yes, the mismatch of kebob-case and snake_case is weird, sorry), so you will want to install using a mechanism that is conducive to making CLI commands available.

# Install this package using
pip install gapic-generator

# Install a version of python that is supported by the microgenerator.
# We use 3.9.12 as an example.
# You may need to install additional packages in order to
# build python from source.
# Setting a 'global' python is convenient for development but may interfere
# with other system activities. Adjust as your environment requires.
pyenv install 3.9.12 && pyenv global 3.9.12

# Install the tool. This will handle the virtualenv for you, and
# make an appropriately-aliased executable.
# The `--editable` flag is only necessary if you want to work on the
# tool (as opposed to just use it).
python -m pip install --editable .

To ensure the tool is installed properly:

$ which protoc-gen-python_gapic
/path/to/protoc-gen-python_gapic

Usage

To use this plugin, you will need an API which is specified using protocol buffers. Additionally, this plugin makes some assumptions at the margins according to Google API design conventions as described in AIPs, so following those conventions is recommended.

Example

If you want to experiment with an already-existing API, one example is available. (Reminder that this is still considered experimental, so apologies for this part being a bit strange.)

You need to clone the googleapis repository from GitHub:

$ git clone https://github.com/googleapis/googleapis.git

It is possible to generate libraries for most (possibly all) APIs described here. The API we use as an example is the Google Cloud Vision API, available in the google/cloud/vision/v1/ subdirectory. This will be used for the remainder of the examples on this page.

You will also need the common protos, which define certain client-specific annotations. These are in the api-common-protos repository. Clone this from GitHub also:

$ git clone https://github.com/googleapis/api-common-protos.git

Compiling an API

Compile the API into a client library by invoking protoc directly. This plugin is invoked under the hood via. the --python_gapic_out switch.

# This is assumed to be in the `googleapis` project root, and we also
# assume that api-common-protos is next to it.
$ protoc google/cloud/vision/v1/*.proto \
    --proto_path=../api-common-protos/ --proto_path=. \
    --python_gapic_out=/dest/

Note

A reminder about paths.

Remember that protoc is particular about paths. It requires all paths where it expects to find protos, and order matters. In this case, the common protos must come first, and then the path to the API being built.

Generating Samples

In addition to generating client libraries, the generator can also create standalone executable code samples.

The user can specify individual sample config files or can pass paths to directories that contain sample configs. Directories are searched recursively, and any file that is not a sample config is ignored.

A full description of the sample config, generated manifest, and generated samples is outside the scope of this documentation. We will provide links to such documentation when it is ready.

Samples and manifests are always generated in a ‘samples’ subdir of the destination directory.

# Multiple sample paths or directories can be passed simultaneously by duplicating
# the 'samples' option. Options are comma delimited.
# If no 'samples' option is passed, the generator does not generate a manifest.
$ protoc path/to/api/protos/*.proto \
              --proto_path=../api-common-protos/ \
              --proto_path=. \
              --python_gapic_opt="samples=sample_config.yaml,samples=sample_dir/" \
              --python_gapic_out=/dest/

Verifying the Library

Once you have compiled a client library, whether using a Docker image, local installation or bazel, it is time for the fun part: actually running it!

Create a virtual environment for the library:

$ virtualenv ~/.local/client-lib --python=`which python3.7`
$ source ~/.local/client-lib/bin/activate

Next, install the library:

$ cd dest/
$ pip install --editable .

Now it is time to play with it! Here is a test script:

# This is the client library generated by this plugin.
from google.cloud import vision

# Instantiate the client.
#
# If you need to manually specify credentials, do so here.
# More info: https://cloud.google.com/docs/authentication/getting-started
#
# If you wish, you can send `transport='grpc'` or `transport='http'`
# to change which underlying transport layer is being used.
ia = vision.ImageAnnotatorClient()

# Send the request to the server and get the response.
response = ia.batch_annotate_images({
    'requests': [{
        'features': [{
            'type': vision.Feature.Type.LABEL_DETECTION,
        }],
        'image': {'source': {
            'image_uri': 'https://images.pexels.com/photos/67636'
                         '/rose-blue-flower-rose-blooms-67636.jpeg',
        }},
    }],
})
print(response)