Setting Up and Connecting To Your Google Cloud Account

View on TensorFlow.org Run in Google Colab View on GitHub Download notebook Kaggle logoRun in Kaggle

This guide is to help first time users set up a Google Cloud Platform account specifically with the intention to use tensorflow_cloud to easily run training at scale on Google Cloud AI Platform. TensorFlow Cloud provides APIs that allow users to easily go from debugging, training, tuning Keras and TensorFlow code in a local or kaggle environment to distributed training/tuning on Cloud.

1. Sign up for Google Cloud Platform

To start go to https://cloud.google.com/ and click on “Get Started For Free". This is a two step sign up process where you will need to provide your name, address and a credit card. The starter account is free and it comes with $300 credit that you can use. For this step you will need to provide a Google Account ( i.e. your Gmail account) to sign in.

After completing the sign up process you will be redirected to Google Cloud Platform welcome page. click on the "Home" tab and make a note of your Project ID and Project number. (see Identifying projects)

GCP_PROJECT_ID = 'YOUR_PROJECT_ID'
PROJECT_NUMBER = 'YOUR_PROJECT_NUMBER'

2.Enable Google Cloud SDK and Authenticate your notebook

Now that you have set up your project we can continue with the rest of the configuration steps directly from this notebook. There Three types of notebook, this step (step #2) is slightly different for each notebook, the rest of the steps (step #3 to #6) are the same for all notebooks.

  • 2.1. Auth for Kaggle notebooks
  • 2.2. Auth for Colab notebook
  • 2.3. Auth for Cloud AI Notebooks - Not supported.

2.1 Setup Auth for a Kaggle Notebook

If you are using a Kaggle notebook you will need to repeat this step for any new notebook that you use with this project. In your notebook click on Add-ons -> Google Cloud SDK and follow the instructions on the prompt. Then run the following command to add your auth credentials to this notebook.

import sys
if "kaggle_secrets" in sys.modules:
    from kaggle_secrets import UserSecretsClient
    UserSecretsClient().set_gcloud_credentials(project=GCP_PROJECT_ID)

2.2 Setup Auth for a Colab Notebook

If you are using a Colab notebook you will need to repeat this step for any new notebook that you use with this project. Run the following command to add your auth credentials to this notebook.

import sys
if "google.colab" in sys.modules:
    from google.colab import auth
    auth.authenticate_user()

Next step is to set up the billing account for this project. Google Cloud Creates a project for you by default which is called “My First Project”. We will use this default project. Use your Project ID (from step 1) to run the following commands. This will show you your Billing Account_ID, make a note of this for the next step.

gcloud beta billing accounts list

Use your Billing Account_ID from above and run the following to link your billing account with your project.

Note if you use an existing project you may not see an Account_ID, this means you do not have the proper permissions to run the following commands, contact your admin or create a new project.

BILLING_ACCOUNT_ID = 'YOUR_BILLING_ACCOUNT_ID'

!gcloud beta billing projects link $GCP_PROJECT_ID --billing-account $BILLING_ACCOUNT_ID

4. Enable Required APIs for tensorflow-cloud in your project

For tensorflow_cloud we use two specific APIs: AI Platform Training Jobs API and Cloud builder API. Note that this is a one time setup for this project, you do not need to rerun this command for every notebook.

gcloud services --project $GCP_PROJECT_ID enable ml.googleapis.com cloudbuild.googleapis.com

5. Create a Google Cloud Storage bucket

We will use this storage bucket for temporary assets as well as to save the model checkpoints. Make a note of the name of the bucket for future reference. Note bucket names are unique globally.

BUCKET_NAME = 'YOUR_BUCKET_NAME'

GCS_BUCKET = f'gs://{BUCKET_NAME}'
!gsutil mb -p $GCP_PROJECT_ID $GCS_BUCKET

Create a service account for HP Tuning jobs

This step is required to use HP Tuning on Google Cloud using CloudTuner. To create a service account and give it project editor access run the following command and make a note of your service account name.

SERVICE_ACCOUNT_NAME ='YOUR_SERVICE_ACCOUNT_NAME'

SERVICE_ACCOUNT_EMAIL = f'{SERVICE_ACCOUNT_NAME}@{GCP_PROJECT_ID}.iam.gserviceaccount.com'
!gcloud iam --project $GCP_PROJECT_ID service-accounts create $SERVICE_ACCOUNT_NAME
!gcloud projects add-iam-policy-binding $GCP_PROJECT_ID \
    --member serviceAccount:$SERVICE_ACCOUNT_EMAIL \
    --role=roles/editor

The default AI Platform service account is identified by an email address with the format service-PROJECT_NUMBER@cloud-ml.google.com.iam.gserviceaccount.com. Using your Project number from step one, we construct the service account email and grant the default AI Platform service account admin role (roles/iam.serviceAccountAdmin) on your new service account.

DEFAULT_AI_PLATFORM_SERVICE_ACCOUNT = f'service-{PROJECT_NUMBER}@cloud-ml.google.com.iam.gserviceaccount.com'

!gcloud iam --project $GCP_PROJECT_ID service-accounts add-iam-policy-binding \
--role=roles/iam.serviceAccountAdmin \
--member=serviceAccount:$DEFAULT_AI_PLATFORM_SERVICE_ACCOUNT \
$SERVICE_ACCOUNT_EMAIL

Congratulations !

You are now ready to run tensorflow-cloud. Note that these steps only need to be run one time. Once you have your project setup you can reuse the same project and bucket configuration for future runs. For any new notebooks you will need to repeat the step two to add your Google Cloud auth credentials.

Make a note of the following values as they are needed to run tensorflow-cloud.

print(f"Your GCP_PROJECT_ID is:       {GCP_PROJECT_ID}")
print(f"Your SERVICE_ACCOUNT_NAME is: {SERVICE_ACCOUNT_NAME}")
print(f"Your BUCKET_NAME is:          {BUCKET_NAME}")

Your GCP_PROJECT_ID is:       YOUR_PROJECT_ID
Your SERVICE_ACCOUNT_NAME is: YOUR_SERVICE_ACCOUNT_NAME
Your BUCKET_NAME is:          YOUR_BUCKET_NAME