Cloud

PyTorch is an open source machine learning framework, primarily developed by Meta (previously Facebook). PyTorch is  extensively  used in the research space  and in recent years it has gained immense traction in the industry due to its ease of use and deployment. Vertex AI, a fully managed end-to-end data science and machine learning platform on Google Cloud, has first class support for PyTorch making it optimized, compatibility tested and ready to deploy. 

We started a new blog series – PyTorch on Google Cloud – to uncover, demonstrate and share how to build, train and deploy PyTorch models at scale on Cloud AI Infrastructure  using GPUs and TPUs on Vertex AI, and how to create reproducible machine learning pipelines on Google Cloud . This blog post is the home page to the series with links to the existing and upcoming posts for the readers to refer to. Here are links to the blog posts in this series:

PyTorch on Vertex AI

How To train and tune PyTorch models on Vertex AI: In this post, learn how to use Vertex AI Training to build and train a sentiment text classification model using PyTorch and Vertex AI Hyperparameter Tuning to tune hyperparameters of PyTorch models

How to deploy PyTorch models on Vertex AI: This post walks through the deployment of a Pytorch model using TorchServe as a custom container by deploying the model artifacts to a Vertex Prediction service.

Orchestrating PyTorch ML Workflows on Vertex AI Pipelines: In this post, we show how to build and orchestrate ML pipelines for training and deploying PyTorch models on Google Cloud Vertex AI using Vertex AI Pipelines.

Scalable ML Workflows using PyTorch on Kubeflow Pipelines and Vertex Pipelines:  This post shows examples of PyTorch-based ML workflows on two pipelines frameworks: OSS Kubeflow Pipelines, part of the Kubeflow project; and Vertex AI Pipelines. We share new PyTorch built-in components added to the Kubeflow Pipelines. 

PyTorch/XLA and Cloud TPU

Scaling deep learning workloads with PyTorch / XLA and Cloud TPU VM: This post describes the challenges associated with scaling deep learning jobs to distributed training settings, using the Cloud TPU VM and shows how to stream training data from Google Cloud Storage (GCS) to PyTorch / XLA models running on Cloud TPU Pod slices

PyTorch/XLA: Performance debugging on Cloud TPU VM: Part I: In this first part of the performance debugging series on Cloud TPU, we lay out the conceptual framework for PyTorch/XLA in the context of training performance. We introduced a case study to make sense of preliminary profiler logs and identify the corrective actions.

PyTorch/XLA: Performance debugging on Cloud TPU VM: Part II: In the second part,  we deep dive into further analysis of the performance debugging to discover more performance improvement opportunities.

PyTorch/XLA: Performance debugging on Cloud TPU VM: Part III: In the final part of the performance debugging series, we introduce user defined code annotation and visualize these annotations in the form of a trace.

Train ML models with Pytorch Lightning on TPUs: This post shows how easy it is to start training models with PyTorch Lightning on TPUs with its built-in TPU support.

A few more articles related to PyTorch on Google Cloud

How To train PyTorch models on AI Platform: In this post, learn how to setup PyTorch development environment on Vertex AI Workbench (previously Notebooks) and use AI Platform Training to build and train a sentiment text classification model using PyTorch.

Increase your productivity using PyTorch Lightning: Learn how to use PyTorch Lightning on Vertex AI Workbench (previously Notebooks)

We have more articles coming soon covering PyTorch and Google Cloud AI. 

Stay tuned. Thank you for reading! Have a question or want to chat? Find us here: Vaibhav Singh, Rajesh Thallam, Jordan Totten and Karl Weinmeister.

PHP is used in ~78% of websites, making it a popular language for developers. If your application runs PHP and you want to take advantage of Google Cloud Spanner for its reliability and scalability, this post is for you!

We will highlight the common steps to migrate an existing PHP application to Spanner. The example we work through modifies the e-commerce platform Magento’s Catalog Module to use the Spanner PHP client library. But these principles can be used for any PHP application that makes use of the Spanner PHP library.

What we aim to accomplish

Use the PHP client library to configure a connection to the Spanner database and manage a session pool.

Execute CRUD operations on the configured database in a reliable manner with transactions.

Become acquainted with some useful snippets and queries inspired by the Magento application implementation.

Observe how the snippets can be used in an application, along with the usual CRUD operations.

Before we begin

Consider working through the Magento Codelab that leverages the magento-spanner-port. While the codelab describes the steps required to get the actual integration working, this post goes into more detail to explain what is happening under the hood.

This document shows a number of examples of how to use Magento with Spanner. The examples are taken from the SpannerAdapter and the AbstractDb implementation in the magento-spanner-port. These are the main files of the Magento Spanner adapter implementation.

Setup

Authentication

As the first step, please follow the Authentication guide to ensure the application is authenticated before executing any of the code snippets below.

Creating a session pool and the connection

After successfully authenticating, create a session pool to facilitate the connection between the application and Spanner. Creating a session is an expensive operation, so it is recommended to create a persistent pool of sessions that can be reused. 

The Spanner client library provides a handy way to alleviate this problem by having a cached session pool. More information on using a cached Session Pool and best practices is available here.

With SysVCacheItemPool, you can share the cached sessions among multiple processes. The CreateSessionPool() method creates and returns a session pool object. Each session connects to a single database and a session can only execute one transaction at a time. The variable minSessions can be used to set the minimum number of concurrent sessions that is expected. The pool is initialized with minSessions when it is created. Then if more than minSessions are required, new sessions will be created up until maxSessions is reached.

To learn more about project identifiers in PHP in relation to ftok, please refer to the section of the PHP documentation here.

Connect to the Spanner database

The created session pool can be used to make a connection to Spanner in the connect() method as shown below. The connection object can be used to perform CRUD operations on Spanner.

Warm up the session pool

Keeping the session alive

The maintain method refreshes sessions that would otherwise expire within the next 10 minutes. In some cases, it can also refresh sessions that would otherwise expire in more than 10 minutes in order to distribute refresh calls more evenly. Only minSessions sessions are maintained. Excess sessions are left to expire.

Note: You should set maxSessions high enough to handle peak concurrency with some additional buffer to allow for growth in peak concurrency. Be aware of memory usage of the process, to avoid the process being killed. Additionally, we recommend setting minSessions equal to maxSessions to avoid the latency cost of having to create new sessions when serving requests. This should minimize tail latencies.

Executing Transactions on Spanner 

After setting up the connection and creating the session pool successfully, we are all set to try some Insert, Read, Update, and Delete operations on the Spanner database.

The snippets below showcase executing CRUD operations using transactions, where they will be committed to the database if and only if all the enclosed queries are successful.

Insert

In the insert() method, we use runTransaction to create the sequence of instructions that need to be executed.

We attempt to insert multiple rows using insertBatch, and if all inserts are successful the transaction is committed to the database.

Update

Very similar to insert, updating an existing batch of data can be done using updateBatch in the update() method.

Transactions allow you to rollback your updates in case any update of a row fails to process.

While updateBatch uses the mutations API, updates can also be performed using DML statements, with executeUpdate. The same is true for inserts mentioned previously. Refer to Comparing DML and Mutations for guidelines on when it’s appropriate to use mutations vs. DML. Please keep in mind that Spanner limits the number of mutations that can be performed in a single transaction. To learn more about how the number of mutations is calculated, please review the documentation. To find the mutation count for a transaction and learn how to prevent your transactions from hitting this limit, see the commit statisticsdocumentation. 

Delete

The delete()method deletes the row using transactions to safeguard against partial failed queries.

In the above code snippet, $where contains the WHERE clause condition and $table is the name of the table to delete from. Spanner requires the presence of a WHERE clause to prevent accidental deletion of the entire table.

Reading from the database

Reading from the database is simpler compared to the transaction queries above, as read queries do not require explicit rollback or commit calls. The example fetchAll() method is shown below.

Here $sql is an SQL string that would be used to query for the results over the database.

Running any SQL query on the Spanner database

Queries are made using the execute() function on the Spanner connection from the class’s query() method. 

Closing the connection

The closeConnection() method releases the resources of the Spanner connection. This must be done after all logical operations have been successfully completed by the application, otherwise sessions will be held forever by the previous caller. This will eventually exhaust the session pool, preventing any further requests from being executed.

More Useful Queries

So far, we described examples of basic CRUD operations. Here are a few more interesting operations inspired from the Magento e-commerce implementation using Spanner. 

Convert iterator into array

Copies the iterator object of results from a query into an array to be used by the application. It’s important to note that large result sets will use a significant amount of memory, so best practice would be to limit results of your Spanner query.

Generate a UUID 

Spanner does not support auto-increment fields and recommends using a UUID. This is because incremental primary keys can create hotspots when accessing data due to the manner in which data is distributed across Spanner servers. See Choosing a primary key for more details. You can use the generateUUID() method to generate random UUIDs.

Sanitize SQL query

Since Spanner uses a strict type of formatting the query, you can use the following sanitizeSQL() method to ensure the value matches the Spanner data type. This is only an example function that handles sanitizing integer data types. It can be expanded in your application to handle other sanitization.

Add casting by type to the columns in queries

As we just mentioned, Spanner is strict on the type of the data being read and written. Although Spanner performs implicit casting, sometimes queries fail when the default implicit casting fails. So, to be safe, we can explicitly specify the cast to be applied to the column using a snippet like the one below. More information on the cast() function can be found here.

Putting everything together

Taking a few excerpts from the Magento code lab that you worked through, let’s discuss how we can use our learnings in practice.

Fetching a catalog of items from the database

Fetching all the items from the database can be as simple as:

Here $select references to the SQL query to select from the database, as described in the “Reading from the database” section above.

Considering that Spanner enforces strict types, using the addCast() function we learned about, the entire operation of reading from the catalog of items from the database can be framed as: 

Getting Wishlist or Cart items

In fetching the required items in the catalog for the wishlist, we would need to run a SQL query for items again. This can be done as follows:

We also learned about sanitizing the queries to adhere to Spanner’s strict type requirements. To do this, the simple getData() method should be modified as shown below:

The sanitizeSql() function from the snippets discussed earlier takes care of the strict type formatting mandated on the queries.

Fetching data with conditions

You will need to ensure your query knows the type of the field. Consider the implementation below from the load() function. The  _getLoadSelect() method returns a simple MySQL query where strings and numeric values for field types are treated alike, i.e. as strings.

To ensure that we are meeting Spanner’s type requirements, replace the function _getLoadSelect() with getLoadSelectForSpanner() as shown here:

Where the implementation of getLoadSelectForSpanner() is as follows:

Saving and updating the objects in your Items  

Here we demonstrate the usage of generateUuid(), based on the fact that Spanner strongly recommends the use of random UUIDs instead of incremental numerals in primary keys. 

Consider a non-Spanner implementation as below:

Which can be replaced by the following:

updateObjectInSpanner() and saveNewObjectInSpanner() make use of our generateUuid() snippet explained earlier. 

To save an object:

To update an existing object:

For reference, the implementation of prepareDataForSpannerUpdate() in updateObjectInSpanner() above, is shown below:

Deleting an object

To complete the D in the CRUD, here is an example of safely deleting an object from Spanner in the context of Magento:

In the Magento example, id columns are generally integers. For other applications it may be useful to validate that the value matches the expected Spanner data type.

Wrapping it all up

The basic tools when interfacing with any database are its CRUD operations. This article introduced these operations for Spanner in the context of PHP and Magento, along with other nuances to keep an eye out for. Please refer to the official documentation here, to continue your journey with Spanner and PHP.  

Cloud Spanner is a fully managed relational database with unlimited scale, strong consistency, and up to five 9s of availability. To achieve the powers of consistency, scale and availability by default Cloud Spanner has a built-in mechanism to automatically shard your database and provide a transparent and seamless experience. Spanner is a distributed database and as your database grows, Spanner divides your data into shards called splits. Individual splits can move independently from each other and get assigned to different servers, which can be in different physical locations. For more information, see Database splits.

Spanner splits data based on load and size. That way, splits can be dynamically moved across Spanner nodes to balance the overall load on the database. The more data you insert into Spanner, the more splits are generated.

In the following diagram, there are four nodes (or 4,000 processing units) so four server resources are available in each zone within the instance configuration. Because you have no data in Spanner, when you start writing data you only write to a single server where the leader of a single split is assigned to. Spanner is currently in a cold state.

The following diagram illustrates the split to the other nodes. As the data comes into the system, Spanner starts to split that data to rebalance the load across server resources provisioned for four nodes. Now Spanner is in a warm state.

To achieve the best performance characteristics, it is advisable  to launch your production application when Spanner is in a warm state with splits already balanced across all of the server resources.

Cloud Spanner Warmup

One of the ways you can pre-warm your database is by following these steps:

Create your Cloud Spanner instance and database with your production schema. It is advisable to include all indexes, to proportionally warm them up as well.

Generate load for the critical tables, those that are expected to get heavy traffic and/or are critical to the latency of your production application.

Ensure that the table primary keys you generate for your load are in the same keyspace (have the same statistical properties) as the keys used for production traffic.

Execute the load two days before your application launch. Also, it’s recommended to execute the load for an hour, at or near the expected peak load. The load causes Spanner to create more splits due to load-based splitting. 

After the warmup is complete, it is recommended to delete the rows using Partitioned DML so that the splits remain available for the application launch. It is not recommended to drop the table or indexes.

Generate Load

To create load, you can use the following tool to generate random and synthetic data for a single table. The tool infers the table schema to generate data. The customers can tune the tool to mimic their data size, column size, key ranges and read/write distribution, thereby providing them the expected performance numbers at launch. Also to generate appropriate sized load (in the range of expected production traffic), it is recommended to execute the tool from a GKE cluster.

To reduce network latency, it’s best to place the GKE cluster close to the Spanner Instance (preferably in the same region as your regional instance or the default leader region of your muti-regional instance).

Cloud Spanner Setup

GKE Setup

Executing the Tool

To create an appropriate load on the Cloud Spanner database, edit the gke_load.yaml file to update the number of threads, operations and pods on the GKE cluster and supply the Spanner resources information.

Monitoring

Monitor the throughput and latency graphs on the Cloud console. The tool will eventually be able to generate higher throughput as more splits are created and able to reach & sustain the expected production peak traffic.  

It usually takes about the first ~30 minutes to get stable p99 latencies, and almost an hour to get stable p99.9 latency. The throughput should get near expected throughput in the first ~30 minutes.

Best Practices

Splits are created based on usage and schema of the database, and reusing an existing database may provide different characteristics than a new database. 

Reusing the database by only dropping the table may preserve unrelated splits and cause unexpected behavior. 

Reusing an existing table for warmup should track the random data being written to the database by the tool. One suggested way is to create a nullable  commit_timestamp  column, that will be auto-filled by gcsb. These rows can subsequently be deleted after warmup and the commit_timestamp column can be dropped after warm up too. 

Create the database with the complete production schema, including Indices.

Caveat: The tool currently doesn’t support the following features. 

The warmup tool provides the ability to tune the data such that they are representative of the production workload:

All primary keys must be in the same keyspace as in the production database to create appropriate splits.

Column data can also be configured for size to maintain overall row length equal to the production load

It is recommended that all your Secondary Index (including Interleaved Index) are created before starting the warmup. Indexes will also automatically split based on the data load. It is important to load index primary keys in the same keyspace as the production workload to get the appropriate splits. 

For interleaved tables, warmup process may differ based on the usage of your child tables:

If your child table is expected to have many rows (1000+) under a single parent row, then it is advisable to add rows (approximately in the same order of magnitude as it would be in production) within the child table during the warmup process.

Though if you only have a few rows under each parent row, then warming up the parent table is sufficient.

The warmup process may take several minutes, even up to an hour to stabilize your system. It is recommended to at least run the warmup process for the ballpark estimates mentioned below:

For a Spanner instance with up to 50 nodes, it may take about an hour to warmup and perform at stable QPS and latencies. For instances larger than that, add about 5-10 mins for instances with double the size. Example: 50 node instances take up to 60 mins, 100 nodes may take 70 mins, 1000 nodes will be about 100 mins.

It is also recommended to tune the warmup tool configuration to execute within the recommended CPU threshold. 

Make sure to track and delete the synthetic data created by the tool before your production application is launched. Data can be deleted using the Partitioned DML.  

It is a prerequisite to tune the threads and GKE pods to create peak traffic. Scale the throughput by adding more GKE pods, rather than scaling the number of threads within each run to avoid CPU contention on each pod.

Verification

To ensure that your Cloud Spanner database would achieve the expected throughput of your production application, you should execute a read/write workload after the warmup. This workload will continue to create further splits, if needed. More importantly, it will provide insight into latency and throughput for the application launch.

This step can also be used to pre-warm an existing database or table. Customers can specify the specific key ranges for the tables (config) that should be split before the launch.

To generate read/write traffic, edit the gke_run.yaml file, supplying the Spanner resource information, and the expected read and write traffic. The configuration also allows you to configure strong/snapshot reads to mimic the production workload closely.

Also, as an additional correlated metric to verify: the full table scan query should at least have as many executions as the number of nodes within your Spanner Instance. To find the query executions, execute the following in the Cloud Console and examine the plan:

Optionally, you can use Cloud Spanner Key Visualizer to view the key distribution in the generated read/write traffic. If you have configured specific key ranges to warm up, you should be seeing consistent higher numbers of reads or writes in those key ranges showing up as bright bands in the Key Visualizer. If you have configured write workload to an indexed column, you should also see write activities in the indexes.

Conclusion

Spanner is a distributed, globally scalable SQL database service that decouples compute from storage, which makes it possible to scale processing resources separately from storage. This distributed scaling nature of Spanner’s architecture makes it an ideal solution for unpredictable workloads such as online games. Learn how to get started developing global multiplayer games using Spanner, in this whitepaper.