Google Cloud

Now in its third year, the Google Cloud Research Innovators program has invited 37 new participants to work with each other and Google experts to accelerate their most promising projects. The program offers select researchers expanded technical and networking support from Google professionals, access to Google’s latest cloud technologies, and additional Google Cloud research credits. By collaborating across fields, sharing resources, and disseminating knowledge, Research Innovators aim to drive new discoveries and solve problems faster.

This year’s cohort includes researchers in a wide range of fields at North American institutions. They meet once a month with mentors and participate in research conferences and special training days. Here are a few of the Research Innovators solving real-world problems with Google Cloud:

Elisa Chen at the University of Chicago uses Vertex AI Vision to develop a model that recognizes faces in illustrations in order to examine race, age, and gender representations in award-winning children’s books in the U.S.

Swapneel Mehta at New York University uses Google Cloud’s machine learning and data processing tools to analyze disinformation online.

Christopher Weber at the University of Arizona uses BigQuery and Data Studio to analyze voter registration data and visualize political behavior.

​​Guangming Zheng at the University of Maryland develops algorithms to identify cyanobacteria in freshwater lakes through satellite imagery, which can help assess threats to human health and ecosystems.

Previous Research Innovators have already generated scientific breakthroughs and had broad social impact: Tapio Schneider developed a ClimateMachine to simulate cloud cover, Richard Fernandes designed an app to monitor Canada’s leaf canopy, and Christoph Gorgulla accelerated drug discovery for COVID-19 treatments. Some of these innovators are now mentoring this year’s cohort to further advance the program’s goals and pass along its benefits.

If you’re a researcher interested in exploring the benefits of the cloud for your projects, apply here for access to the Google Cloud research credits program in eligible countries. To engage with the latest challenges and solutions in research and technology, sign up for NEXT, Google’s global exhibition of inspiration, innovation, and education in San Francisco August 29-31.

Mobile photography has become ubiquitous over the past decade, and it’s now easier than ever to take professional quality photos with the push of a button. This has resulted in explosive growth in the number of photo and video captures, and a huge portion of these photos and videos contain private, cherished, and beloved memories — everything from small, everyday moments to life’s biggest milestones. Google Photos aims to be the home for all these memories, organized and brought to life so that users can share and save what matters. 

With more than one billion users and four trillion photos and videos — and with the responsibility to protect personal, private, and sensitive user data — Google Photos needs a database solution that is highly scalable, reliable, secure, and supports large scale data processing workloads conducive to AI/ML applications. Spanner has proved to be exactly the database we needed.

A picture says a thousand words

Google Photos offers a complete consumer photo workflow app for mobile and web. Users can automatically back up, organize, edit, and share their photos and videos with friends and family. All of this data can be accessed and experienced in delightful ways thanks to machine learning-powered features like search, suggested edits, suggested sharing, and Memories. With Photos storing over 4 trillion photos and videos, we need a database that can handle a staggering amount of data with a wide variety of read and write patterns. 

We store all the metadata that powers Google Photos in Spanner, including both media-specific and product-specific metadata for features like album organization, search, and clustering. The Photos backend is composed of dozens of microservices, all of which interact with Spanner in different ways, some serving user-facing traffic, and others handling batch traffic. Photos also has dozens of large batch-processing Flume pipelines that power our most expensive workloads: AI/ML processes, data integrity management, and other types of full account or database-wide processing.

High level architecture for media processing in Google Photos using Spanner

Despite Google Photos’ size and complexity, Spanner has a number of features that make our integration easy to maintain. Thanks to Spanner’s traffic isolation, capacity management, and automatic sharding capabilities, we are able to provide a highly reliable user experience even with unpredictably bursty traffic loads. Balancing our online and offline traffic is also manageable thanks to Spanner’s workload tunable replication capabilities. 

Photos enables users to access all of their photos at any time, reliably across the globe. Photos relies on Spanner to automatically replicate data with 99.999% availability. Spanner’s sharding capabilities give us low latency worldwide, help us smooth our computational workloads, and make it easy for us to support the ever increasing set of regulatory requirements concerning data residency.  

The system has to be reliable and available for user uploads, while simultaneously ensuring that ML-based features not only perform well, but also don’t impact interactive traffic. Spanner’s sharding flexibility allows both these use cases to be satisfied in the same database. We have read-only and read/write shards to separate these use cases. We need to serve our active online users quickly because we know they expect their photos to be instantaneously displayed and shareable.

Photos also has strict consistency and concurrency needs. That’s not surprising when you consider the variety of first- and third-party clients that upload media, processing pipelines performing updates, and various feature needs – many of which involve cross-user sharing. It’s Spanner’s high write throughput, consistency guarantees, and resource management tools that have allowed Photos to build and scale these features and pipelines by 10x with minimal re-architecture. Our use of Spanner has proven Spanner’s ability to scale rapidly without compromise — something rare in traditional, vertically scalable SQL databases. 

Equally as important, Spanner has significantly increased our operational efficiency. We now save a lot of time and energy on tactical placement, location distribution, redundancy, and backup management. Replica management is a simple matter of configuration management, and we rely on Spanner to manage the changes. In addition, automated index verifications, automatic sharding, and guaranteed data consistency across all regions, save us a lot of manual work.

Trust paints the whole picture

Our users entrust us with their private and precious data, and we take that responsibility very seriously. Privacy, security, and safety are incredibly important to Google Photos — they are core principles that are considered in every feature and user experience that we build. Spanner’s secure access controls help significantly by eliminating unilateral data access, managing the risk of internal or external data breaches, and ensuring that data privacy is respected throughout our backend. 

Reliability and trust are the cornerstones of Google Photos. It’s critical that users can access their data whenever they want it, and that fundamental product features like backup and sharing remain highly available even during peak load (holidays, for example). The Photos team continues to heavily focus on reliability improvements to ensure that we’re delivering the experience that our users have come to expect from Google. Thanks to Spanner’s ongoing investment in this area, Photos has been able to continuously raise this bar — which is particularly notable given Photos’ own rapid growth rate. Running multiple replicas is a key aspect of how our system runs reliably, and Spanner’s strong external consistency features and continuous index verifications ensure that data remains correct. In addition, Spanner offers robust backup and recovery systems which provide us even more confidence that our datastores will remain correct and complete. 

Picture perfect

The numbers speak for themselves. Spanner supports a staggering amount of traffic across many regions, over a billion users, and metadata for more than four trillion images. We’ve already experienced 10x growth since launching our Spanner database, and we’re confident that Spanner can support another 10-fold increase in the future. Going forward, we’re confident in Spanner’s robust, easy-to-use nature to help us scale to the next billion users and drive even more incredible experiences for our users. 

Learn more

Read the blog “SIGOPS Hall of Fame goes to Spanner paper — here’s why that matters” by Chris Taylor, Distinguished Software Engineer, on the recent award and the evolution of Spanner.

Learn more about Cloud Spannerand create a 90-day Spanner free trial instance.

Vodafone is one of the largest telecommunications companies in the world, and committed to building next-generation connectivity and a sustainable digital future. Last year, we discussed AI Booster, an AI/ML platform that we built that runs on Google Cloud, and how we use it to optimize customer service, improve customer loyalty, and provide better product recommendations. Besides the benefits it brings to Vodafone’s overall operations, AI Booster is also interesting because of the way that it uses CI/CD and automation technologies. We believe this pattern has the potential to be applied to a wide range of use cases.

High-level architecture

AI Booster provides an opinionated abstraction around Google Cloud services (especially Vertex AI), ensuring automation and security guardrails. At the same time, we worked to drive efficiency, reduce operational overhead and deployment time in AI Booster by automating steps such as CI/CD and code mirroring, which significantly reduced our application delivery times while helping to keep data secure.

AI Booster consists of several projects, and connects to various core Vodafone services. Let’s take a closer look.

The Datahub

An important part of the overall AI Booster environment is the Datahub, Vodafone’s data lake, one of our most precious and valuable data vaults. Built on BigQuery, the Datahub is divided into local markets, and is the source for all the data used in our AI/ML use cases. Access to it is vetted by each data segment’s respective owners, and each local market uses a different project to enforce additional segregation. Vodafone’s Data Science teams get read-only access to the production data they need for a particular use case. 

AI Booster

AI Booster itself consists of five projects per use case, each with a distinct purpose. Three of those projects are fairly typical environments:

LAB: A development environment where data scientists can create and iterate their models. Scratch storage will be used in R/W and wiped every 14 days for compliance reasons.

STAGING: The staging/pre-production staging area. It has the same storage setup as the LAB environment.

PROD: The production environment with R/W access to the production Datahub.

In addition, there are two auxiliary Google Cloud projects, which are used for build purposes:

BuildSTAGING: Cloud Build, from LAB to STAGING

BuildPROD: Cloud Build, from STAGING to PROD

The entire setup including all environments is created upon request in Vodafone’s workflow tool. The workflow integrates privacy and responsible AI governance processes and requires approval from respective stakeholders for the use case or project before deploying the resources. “Once the approval is granted, the workflow tool triggers the creation process using Terraform, spinning up all Google Cloud projects, service accounts, VPC/networking, etc. and leveraging Google Cloud Build. The architecture has security built in, e.g., using VPC-SC, CMEK, Cloud NAT, and there is no access to public internet,” says Omotayo Aina, Solution Architect at Vodafone Group Technology.

Central mirror and repositories 

Whenever Vodafone data scientists or ML engineers need to access any libraries, packages, or containers, they actually receive them from a central mirror. The central mirror is a key component of the AI Booster architecture, providing vetted artifacts (from Artifact Registry) and making for fast and easy retrieval. The central mirror consists of:

PYPI mirror: Python libraries local mirror

hub.docker.com mirror that pulls from public docker repos (such as Mongo)

Container store: Containers you can use to spin up notebooks or Vertex AI pipelines

For added security, all artifacts are scanned, and any “compromised” versions are flagged centrally, preventing developers from pulling vulnerable artifacts. Projects that require vulnerable (flagged) artifacts must look for an alternative artifact version with no vulnerabilities. If it’s not feasible to replace an artifact, we perform a review of the residual risk and mitigate it before granting an approval. A NAT/firewall configuration prevents developers from using the original repositories outside the Vodafone network, ensuring they only use mirrored artifacts.

AI Booster at work: Predicting churn in two different markets

Vodafone operates in 17 countries, mostly in Europe. Recently, Vodafone Group’s Big Data & AI team was tasked with building customer churn prediction models for some of these countries, but found it was a complex task. 

“Each local market has its own set of requirements, and different data dimensions are available per country. Yet, we want to quickly replicate our work across markets,” said Ali Cabukel, a Vodafone Machine Learning Engineer. “AI Booster enables us to achieve this much more easily.” 

To understand the AI Booster architecture better, and how it helped improve our operations, let’s start with a real use case: predicting customer churn for one local market, and then replicating it for another local market.

1. LAB environment: development

A data scientist (DS) starts off in the LAB environment. Here they have access to:

Local Datahub (BigQuery) with access to Customer Analytical Records (CAR), which are wide tables with bills, calls, data usage and other customer data specific to a local market

A private GitHub repository for each use case

A number of Google Cloud tools within AI Booster, e.g, Vertex AI, BigQuery, and BigQuery ML

Data scientists spend their time with data preparation, pre-processing, feature selection, and building models using frameworks like BQML, Spark ML (serverless), XGBoost, LightGBM, scikit-learn, TensorFlow, and Google’s AutoML. All code is regularly checked into a GitHub repository as a best practice.

To accelerate the work, there is a central template repository that contains Cloud Build configuration files, Kubeflow Pipelines (KFP), pre-built (reusable) KFP components, Build Triggers, unit tests, payload files, queries, and dependency requirements files. Here is a sample repository that’s publicly available from Google Cloud.

As we refine the use case, the ability to perform several parallel experiments with data becomes a priority; for that, data scientists and machine learning engineers need to be able to quickly create a training pipeline in Vertex Pipelines, leveraging the assets provided in the template repository.

2. From LAB to STAGING

Once the LAB environment is considered stable, the KFP pipeline is promoted to STAGING, so data scientists and machine learning engineers can start integration testing any changes that they first implemented in the LAB environment. 

How does this happen? The BuildSTAGING Cloud Build pipeline transfers code and pipelines to the STAGING environment, where data scientists check for data errors and pipeline behavior, and set up proper triggers in Cloud Scheduler, Pub/Sub, and Cloud Functions. To deploy the Kubeflow pipeline to STAGING, CI/CD Github triggers are set up on the BuildSTAGING Google Cloud project using Cloud Build. Specifically, 

pr-checks tests the Pull Request for linting, unit tests, and catchable errors

terraform-plan verifies the Terraform desired config “compiles”

Then, after some back and forth where code is linted, tested, and finally approved, the code is actually merged. That, in turn, sets off another two triggers: 

terraform-apply: this trigger actually applies the infrastructure changes via Terraform, including time-based scheduling (cron jobs) to Cloud Scheduler on the STAGING/PROD project

release: the release trigger is fired with a new GitHub release tag (versioning). This trigger is responsible for compiling the Kubeflow pipelines, transferring all resources to a Cloud Storage bucket and conducting end-to-end tests.

3. From STAGING to PROD

Finally, the model is pushed to PROD and is now able to read/write data to the Vodafone DataHub. How? Similar to the previous step, the BuildPROD project promotes the pipeline structure from STAGING to PROD. The structure between STAGING and PROD is identical, with the only difference that the production environment has write access to the production DataHub, whereas the previous stages only have r/w access to an ephemeral local copy.

4. And back to LAB…

Let’s say that a data scientist now wants to do similar work for another local market. While the data might look different and some experimentation is required, there are likely many similarities between the two, including the pipeline. As such, moving to STAGING and PROD will be much faster. The overall process will look like this:

5. Change management

AI Booster uses TensorFlow Data Validation to identify data skew between training and prediction data. This tool is very useful for catching data anomalies with respect to data distributions. If needed, the data scientist goes back to LAB to fix the problem; changes are then propagated quickly to production. You can read a more in-depth view of how we use TensorFlow Data Validation in our data contracts in this TensorFlow blog post.

Results and improvements

As a result of using AI Booster, we see multi-fold improvements – most notably efficiency gains:

Reduced time to market (5x): We found a robust, repeatable and trustworthy toolchain to allow our data scientists to be successful on the platform, without having to compromise between data security and flexibility in experimentation

Improved maintenance and data quality: By leveraging more automation and standardization, we were able to ensure common quality standards and thus also reduce maintenance efforts over the lifecycle of a use case.

More flexibility: Deployment frequency has increased by ~5x (from one pipeline in a month to five in a month). This gives us the flexibility to react quickly to changes and work in an agile way.. In many cases, we’ve removed extensive reviews and alignments from the process entirely, as security and compliance are baked in.

Improved compliance processes: Integrating privacy and responsible AI requirements early on in the development processes helps to ensure a “by design” approach and address the topics in the right time of the development lifecycle.

At Vodafone, our main objective is to empower data science teams to focus on customers. Overall, the lightweight AI Booster wrapper makes it easy for central or distributed teams to evolve the platform and handle maintenance efficiently. We are excited to continue evolving AI Booster in the future..

Thanks to Max Vökler, Technical Account Manager, for his contributions to this post.

Building the backend for a fast-paced competitive multiplayer game is a complex task. There are many moving parts to coordinate, and the product and solution landscape is practically endless. There are a myriad of managed products, supporting open source solutions and more that need to be evaluated, experimented with and implemented, and the path to using all of them in conjunction with one another is often not clear.

To solve this problem, and kickstart your journey on building your own multiplayer game, a multidisciplinary team of Google Cloud for Games experts banded together to create a global scale, integrated demo of a fast paced multiplayer game. The aim of this demo is to show the  technologies and techniques we at Google Cloud have seen work in production to host, scale and orchestrate game backends for multiplayer games – from services and data storage all the way through to dedicated game servers.

This includes:

Game Engine

Unreal Engine 5 for the game client and dedicated server code.

A custom Go game launcher for client side authentication.

Open Source

Terraform, for infrastructure as code.

Agones for hosting and scaling dedicated game servers.

Open Match for match making our global player base.

Google Cloud

Cloud Build and Cloud Deploy for Continuous Integration and Deployment.

GKE Autopilot for hosting the backend microservices.

Globally distributed GKE Autopilot and Standard clusters running for hosting Agones orchestrated Unreal dedicated game servers

Anthos Service Mesh for cross cluster service discovery and communication.

Cloud Spanner for storing the player in-game data.

We’ll also share some tips and tricks the team of people working on this project learnt along the way so you can turbocharge your development journey even more!

Droidshooter

We deliberately built a very simple multiplayer game called “Droidshooter” to highlight and make easily discoverable the server side technologies and techniques we took advantage of during the development of this demo.

In this game you fly a small spaceship against another two players, while attempting to get their health to zero by firing bursts of energy against the other ships. For each successful destruction of a rival spaceship, you receive a point. The player with the most number of points at the end of thirty seconds wins the round!

Architecture overview

Before we jump into the details of each component, let’s look at an architecture overview:

Since this is a game that is targeted at a global audience, we also have a global infrastructure footprint.

Our general backend services for our game are hosted in us-central1, since they aren’t latency sensitive, and consolidating them into a single region and cluster makes managing them much easier. Since player connection to the multiplayer, dedicated game servers are latency sensitive, we have them on multiple Kubernetes clusters in multiple regions around the world to ensure that they are close to our players locations.

Game state flow

Launching the game client

In the above architecture, each player will start an instance of the Game Launcher. This provides authentication via Google login, as well as options for launching the game.

At log in, a profile record is created if it doesn’t exist in the database. From there, the profile information is retrieved from the profile service and displayed here in the launcher (That’s one of our names above!) as well as in the game client.

Clicking “Open Droidshooter” will start the game client, which will request a game through the frontend service, providing an authenticated endpoint for almost all communications to hosted backend services for the game client, as well as the previously mentioned launcher.

Match making

From here, the client will need to request the global set of “ping locations” in each of the regions that our game servers will be hosted in, since our game requires low latency for an optimal, real time experience. Therefore we need to actively measure ping latencies from each region from the player’s location, so that they can be matched to players of similar latency from where they are situated, and ignore regions where latency is too high.  Unfortunately, you can’t just send players to only the closest region to them, as that splits your player base, and results in an inability to match players that would otherwise be able to play a game together.

Once the game client has utilized each of the latency measurement endpoints in each region and recorded the player’s latency to each region, it sends this data to the matchmaking system (through the authenticated frontend) requesting to be grouped with other players so they can play a game! In Droidshooter, we have a very simple matchmaking system, where our matchmaker looks to match three players together that have the closest latency to each other.

Allocating a game server

Once a group is formed to play a game, the matchmaker requests a dedicated game server from the game server orchestration system in the region in which it has determined the best place to place these players. Since all these players need to have the same experience, the dedicated game server’s job is to simulate the game world state, and communicate this information to all connected players in real time, so they can all enjoy the fun experience of playing Droidshooter.

Therefore the matchmaker passes back to the game clients (through the frontend) the address and port each client will need to connect directly to, since the game state is an in-memory simulation, and then, they play Droidshooter!

Final cleanup

Once the game is complete, the game server shuts itself down, and thanks to our game server fleet management system, a new one is spun up right away for new players to play on.

Ecosystem breakdown

Let’s look at each of the products and open source projects we used to implement the above flow, and why we chose each of them:

Terraform

When getting started with this project, the first thing we did was decide that we wanted to have an infrastructure as code set up for automating the creation of the infrastructure we would be running the backend for our game on.  We decided on Terraform, an open-source infrastructure as code software tool that enables you to safely and predictably create, change, and improve infrastructure through a declarative configuration language.

We chose this tool, as it is widely adopted and meant that each of the people developing on the project would have the same infrastructure setup as each other, our final end users could easily spin up the same infrastructure themselves using tools they were already likely familiar with.

Tips and Tricks

Make liberal use of Terraform variables throughout your code! Thanks to our set of configurable variables, end users could choose to spin up a full global configuration of the project, or aim for a much smaller, and cheaper to run, configuration, possibly for development purposes.

GKE Autopilot and Standard

GKE Autopilot is a fully managed Kubernetes experience where you only pay for the Pods you run, and don’t have to worry about the underlying infrastructure that hosts it.  It was a perfect fit for all our services, such as custom ones like our profile and front end service as well as the Open Match platform services (more on that later). The truly delightful thing about developing on Autopilot was that as a team we never had to worry about new services that were being built and if there was capacity on the GKE cluster that they were going to run on – we could just deploy knowing that Autopilot would take care of the infrastructure.

We actually host our game server containers on both GKE Standard and Autopilot in this demo. While you probably would want to settle on a single platform in production, we wanted to highlight that if the ease of use of Autopilot fits your requirements for your game server workloads then it works perfectly with this entire stack, but if your game server requires very specific tuning of the underlying GKE standard nodes – then GKE standard is also a great solution.

Tips and tricks

Start with Autopilot and only move over to GKE standard if you need to. If you don’t have a reason to move to GKE standard, the ease of use of Autopilot can save you a lot of time. Since it’s all Kubernetes underneath, migration between platforms is very straightforward.

Agones

Agones is an open-source game server orchestration platform originally co-created by Ubisoft and Google Cloud, but now with over 200 contributors from a variety of game studios. Built recognizing that Kubernetes needed game-specific optimizations, it teaches Kubernetes about the lifecycle of game server workloads, and ensures that player experiences aren’t interrupted while players are playing on them. Therefore our decision to use Agones was pretty straightforward as this meant we didn’t have to worry about game server lifecycle management or infrastructure usage optimisation – Agones would take care of all that for us.

Agones also provides us with a set of APIs and SDKs that make it easy to integrate our demo with the platform. This includes features like game state management and player management, which significantly decreases the time needed to implement these features from scratch. 

Tips and tricks

Take advantage of the Terraform modules Agones comes with for you to use to create both GKE standard and autopilot clusters. You could create your own from the cluster creation instructions – but why go to all that effort?

Open Match

Open Match is another open-source tool for multiplayer games that was originally developed in collaboration between Google Cloud and Unity Technologies.  It is a matchmaking framework that helps game developers build scalable and extensible matchmaking systems. It provides a set of tools and APIs that game developers can use to create their own matchmaking logic, while Open Match handles the underlying infrastructure to support a global player base.

We primarily chose this platform because it allowed us to have full customization over our match making logic, but also provided us with the ability to scale to whatever traffic we want to drive this multiplayer game demo to handle.

Tips and Tricks

While Open Match bundles a version of Redis hosted within the same Kubernetes cluster it is deployed on, take advantage of MemoryStore for Redis for a managed Redis offering on Google Cloud that includes high availability, failover, patching, and monitoring.

We recommend hosting Open Match on GKE Autopilot, since you only pay for the Pods you are running, and no longer have to concern yourself with node infrastructure – making Open Match much easier to manage and maintain.

Unreal Engine 5

Unreal Engine is a great choice for building multiplayer games, as it has a long history of use within the industry, provides fantastic networking tools and also is used by many developers in production for their multiplayer games! So since we were looking to match production workloads with this sample, it seemed like an easy decision.

One of the many benefits of using Unreal Engine 5 is its seamless integration with Agones, which can be achieved both via C++ and Blueprints. With the Agones SDK integrated into Unreal Engine 5, developers can easily manage the game server’s lifecycle, configuration, metadata and player tracking. You can download the source and build the plugin directly from here: https://github.com/googleforgames/agones/tree/release-1.30.0/sdks/unreal/Agones

Tips and tricks

To query the latency times to each region we are hosting game servers in, our Unreal Engine 5 based client also needs to send some UDP packets to Agones’ Latency Testing Services.. This can be a little tricky to find the documentation for, but to achieve that, we can use Unreal’s FUDPPing API,  to send an ICMP Ping packet over UDP, like so: FUDPPing::UDPEcho()

The Unreal Engine container ecosystem for interacting with the engine in the cloud – particularly building container images of dedicated game servers, is really well implemented and has a very smooth developer experience – strongly recommend taking it for a spin if you haven’t!

Cloud Build and Cloud Deploy

Cloud Build and Cloud Deploy are two managed Google Cloud Platform products that can be used together to automate the Continuous Integration and Continuous Deployment of applications that also have strong integration with GKE standard and Autopilot. For the Global Scale Game Demo, we are using them for both creating the artifacts and images for the backend services as well for the Unreal Engine 5 dedicated game servers, using the aforementioned Unreal Engine container system. 

Cloud Deploy was also particularly chosen because of its ability to arbitrarily group multiple clusters together as deployment targets. This means we can group our clusters per region, and do regional rollouts of our clusters.

This means we can do test deployments in smaller population areas, such as Australia, and assuming all things go to plan, move out to large and large population sets.

Tips and Tricks

If the generated YAML that is being applied to your cluster through Cloud Deploy results in more than 262144 bytes, use the –server-side  kubectl flag, and you may also want to combine it with –force-conflicts flag to handle any Kubernetes server side apply ownership issues.

If you have a Deployment that may fail several times before coming to a healthy state, take advantage of the tolerateFailuresUntilDeadline feature of Cloud Deploy, which allows the system to tolerate failures up until the specified deadline. This is particularly useful for Istio and/or Anthos Service Mesh, which is covered in later sections.

Artifact Registry

The artifacts that are built and deployed by Cloud Build and Cloud Deploy are stored in an Artifact Registry container repository, which gives us the added benefit of faster spin up times for our backend service and dedicated game server images via image streaming, as well as vulnerability scanning and more.

Anthos Service Mesh

To build a full global scale game we needed a way to communicate between our set of GKE clusters that are distributed around the world. This included our backend services clusters as well as our Agones clusters. This is particularly important to our Open Match based matchmaker, as it needs to route requests to allocate game servers from Agones powered GKE clusters in specific regions around the world.

Thankfully, Google Cloud comes with Anthos Service Mesh, a managed Service Mesh offering based on the open source project Istio. This provides us both with DNS based service discovery within clusters as well as the ability to facilitate multi-cluster communications – which made it a perfect fit for our needs.

The ability to use Istio’s VirtualService was key, as it allowed us to have a single DNS entry for all Agones Allocation REST endpoints in our backend service cluster, and provide routing to specific regions using a custom region HTTP header in the request, while also managing access control and security through Workload Identity. This solved a lot of problems for us in one go!

Tips and Tricks

You may need to use Cloud Build to do some Anthos Fleet Management via gcloud, since some operations are not yet supported via Terraform.

If the expected multicluster Service DNS entry is not found within the cluster that needs it, make sure to ensure the namespace and matching Service are also created in that cluster as well. It doesn’t need the implementation, but it does need both the matching namespace and Service to create the DNS entry.

Cloud Spanner

The classic problem with game launches is that you are never sure how much traffic you are going to get. At the same time launches are critically important because if they go badly, it’s extremely hard to recover your player base from the peak you would receive at launch.

To solve those very problems, we chose Spanner for our data storage platform. Since Spanner has both high availability, with a 99.999% uptime SLA for Multi-Region instances, horizontally scalability to match any demand a launch can throw at it, and also strong consistency globally, so you always know that players are seeing accurate in-game information.

Tips and Tricks

It is important to place the GKE clusters as geographically close to Spanner as defined by the instance configuration to reduce request latencies for writes and consistent reads. In this example, we put Spanner in us-central1 because the backend services are also in us-central1.

Liquibase is a great tool for managing your Spanner schema changes and updates, through a yaml configuration file.

Configuring Workload Identity to enable your application connectivity to Spanner has multiple steps, and can be tricky to get right. To solve this we generated the Cloud Deploy yaml files from the Terraform created IAM configuration to ensure both our infrastructure setup and our continuous deployment configuration aligned without manual intervention.

Conclusion

Choosing a set of products to build a multiplayer game out of the myriad of open source tools and products can be overwhelming and time consuming. The aim of the Global Scale Multiplayer Demo is to provide some guidelines, architecture patterns and building blocks that you can use to build your next global scale, production multiplayer game!

We’re excited for the future of the Global Scale Multiplayer Demo. We have big plans for the future to continue to grow and extend this project to show you all the ways you can use Google Cloud to host and scale your game.

If you want to get involved or just have a look at the project:

Github Repository

Install it in your own Google Cloud Project

File an issue for enhancements you would like to see

This is the first installment of a seven-part series covering the key findings from the State of Kubernetes Cost Optimization, a quantitative research report analyzing the cost efficiency of real-world, large-scale, anonymized Kubernetes clusters.

Workload requests are used everywhere in Kubernetes, from scheduling to scaling up and scaling down the cluster, to workload quality of service, to cost allocation, and more. 

It should be no surprise that getting these right is at the heart of building a cost optimized Kubernetes cluster. It is also the first key finding of the State of Kubernetes Cost Optimization report:

Mismanagement of requests can disrupt applications, having a cascading negative impact on your application’s reliability. Let’s review one actionable strategy to help you master this fundamental aspect of Kubernetes cost optimization.

Not setting requests properly can put workloads at risk

In the report, the “At Risk” segment consists of clusters where the sum of actual resource utilization is generally higher than the sum of their workloads’ requested resources. This results in a higher risk of intermittent and hard to debug reliability issues caused by Node-pressure eviction. 

Node-pressure eviction occurs when resources like memory,  reach a capacity threshold  on a node. This pressure state activates the Kubernetes kubelet to start reclaiming resources by abruptly terminating pods. The priority for killing pods is given to those that are using the most resources compared to what they have requested. 

However, BestEffort Pods, which do not explicitly request any resources, have a high chance of being the first to be terminated due to their lack of specific requirements. If the kubelet cannot kill workloads fast enough, the Linux oom_killer – or “out of memory killer” – can also step in and kill containers. This process also accounts for prioritizing in part based on QoS class. 

Workloads with a QoS class of BestEffort, and workloads with a QoS class of Burstable with memory under-provisioned can be the first to be terminated. Errors which could be related to this termination, such as OOMKilled, ContainerStatusUnknown,  Error, and Evicted, can make troubleshooting difficult. 

This is where viewing observability metrics can be particularly useful. For this reason, the report recommends that requests be properly set for critical workloads:

So how can we take action? Auditing clusters for workloads at risk can be difficult – especially if it is shared by multiple teams.  

Introducing the GKE Workloads at Risk dashboard

That is why we are excited to introduce theGKE Workloads At Risk dashboard, a template in Cloud Monitoring that you can use to monitor BestEffort and Burstable workloads at risk for performance or reliability issues. 

You can also manually navigate to the dashboard by: 

In the Google Cloud console, select Monitoring. You can also navigate directly to Cloud Monitoring using the following link.

In the navigation pane, select Dashboards.

In the Sample Library pane, select Google Kubernetes Engine.

Click the Preview button next to the GKE Workload At Risk from the list of dashboards

The “Best Effort Workloads at Risk” section shows workloads that do not have defined CPU or memory requests. If the memory request is undefined, your Pods may be evicted at any time and marked as failed. Similarly, if the CPU request is undefined, your workloads may become unresponsive as the CPU may be throttled to 0.

To avoid this, you should define CPU and memory requests for your workloads. This will ensure your workloads have the resources they need to run properly.

The “Burstable Workloads at Risk” section shows Pods that are using more memory and CPU than they requested. This can cause reliability issues if there is not enough memory available. These Pods, like the BestEffort Pods mentioned earlier, may be killed. Workloads that consume more CPU than requested may be throttled, resulting in poor performance that could be experienced by the end users of your workloads.

Identifying workloads at risk across your entire fleet

For organizations running multiple GKE clusters across multiple projects, we’ve developed a comprehensive and robust solution built atop BigQuery and Looker Studio.

This solution harnesses the power of BigQuery to deliver swift insights across clusters and projects. It helps identify at-risk workloads while also providing recommendations, allowing you to monitor your progress with a historical perspective. You can also customize it for specific needs to realize consistent improvements. 

To deploy this enhanced and customizable solution today, please follow this solution guide.

If you have Kubernetes clusters running elsewhere

The fundamental goal is to identify workloads that may be at risk due to how requests are – or aren’t – configured.  If you’re looking for a lightweight method that works across any Kubernetes cluster, you can evaluate kube-requests-checker, a simple script that lists all user containers without requests for CPU, Memory, or both, in a given cluster.

You can also access an individual workload’s quality of service class by running the following kubectl command.

In conclusion

With these tools, platform teams and application developers can put the first key finding of the State of Cost Optimization report into practice. This means working together to set requests for workloads that have any reliability requirements and building this into your culture. 

You can find the resources discussed in this blog, along with a tutorial to set up example workloads in GKE, below:  

The GKE Workloads at Risk dashboard template

The Right-sizing workloads at scale solution guide

The simple kube-requests-checker tool

An interactive tutorial to get set up in GKE with a set of sample workloads

Download the State of Kubernetes Optimization report, review the key findings, and stay tuned for our next blog post!

BigQuery is a completely serverless and cost-effective enterprise data warehouse that works across clouds and scales with your data. User data is stored in BigQuery tables. Every table is defined by a schema that describes the column names, data types, and other information.

Recently, BigQuery introduced Unenforced Primary Key and Foreign Key Constraints. This post is a deep dive into Unenforced Key Constraints and how they may benefit queries in BigQuery.

Defining Constraints

Users can define constraints on a table when creating a table using the CREATE TABLE statement. Constraints can also be added to an existing table using the ALTER TABLE ADD PRIMARY KEY statement or the ALTER TABLE ADD CONSTRAINT statement. Constraints can be dropped using the ALTER TABLE DROP PRIMARY KEY statement or the ALTER TABLE DROP CONSTRAINT statement.

For example, The inventory table from TPC-DS has a composite primary key using the columns (inv_date_sk, inv_item_sk, inv_warehouse_sk), and 3 foreign keys referencing date_dim, item, and warehouse. The following statement can be used to create the table with the constraints:

To add constraints to an existing inventory table, the following ALTER TABLE statement can be used:

The user must use the NOT ENFORCED qualifier when defining constraints as enforcement is not supported by BigQuery at this time.

Table constraints can be viewed in the BigQuery UI in the Key Column.

The tooltip shows more information about the keys.

Alternatively, users may query the TABLE_CONSTRAINTS, KEY_COLUMN_USAGE and CONSTRAINTS_COLUMN_USAGE INFORMATION SCHEMA views.

Why create Primary Keys and Foreign Keys?

One may wonder “If Key Constraints are not enforced, why should users create them?”

The answer to this question is that query optimizers may use this information to better optimize queries.

Below, we will discuss three query optimizations that leverage Key Constraints: Inner Join Elimination, Outer Join Elimination and Join Reordering.

Inner Join Elimination

Joins are one of the most expensive operations in databases and a lot of query optimization techniques attempt to optimize them. However, nothing beats completely eliminating joins.

Let’s look at the following tables from the TPC-DS schema:

The store_sales table is one of the fact tables in the dataset. It has a composite primary key (ss_item_sk, ss_ticket_number) and it references 9 dimension tables:

ss_sold_date_skreferencesdate_dim(d_date_sk)

ss_sold_time_skreferencestime_dim(t_time_sk)

ss_item_skreferencesitem(i_item_sk)

ss_customer_skreferencescustomer(c_customer_sk)

ss_cdemo_skreferencescustomer_demographics(cd_demo_sk)

ss_hdemo_skreferenceshousehold_demographics(hd_demo_sk)

ss_addr_skreferencescustomer_address(ca_address_sk)

ss_store_skreferencesstore(s_store_sk)

ss_promo_skreferencespromotion(p_promo_sk)

With that information, consider the following query:

The query above only selects columns from the fact table which is joined with the dimension table using columns that form a referential relationship.

Using the constraints, we know that each row in store_sales will have a single match in customer or will have no match if ss_customer_sk was NULL. The query optimizer can then eliminate this join when creating the query plan.

Let’s run the query first without defining the constraints and look at the execution plan:

As we can see, the query contains a single join between the two tables.

Let’s define the constraints and run the query again. We now get the following plan:

The join has been eliminated and we still got the same result. This was only possible due to the presence of the Key Constraints.

Outer Join Elimination

Removing outer joins has fewer requirements. To remove a LEFT OUTER JOIN, the join keys on the right side must be unique and only columns from the left side are selected. To remove a RIGHT OUTER JOIN, the join keys on the left side must be unique and only columns from the right side are selected.

The joins can be eliminated in this case since having unique keys guarantees that each row from the other side will have at most one match.

Consider the following query:

There is no relationship between ss_item_sk and c_customer_sk.

Let’s first run the query without defining PK constraints on the customer table. We can see the join in the query plan.

After defining the constraints, we ran the same query and can see that the join has been eliminated.

Why would a user write the queries above? Shouldn’t the user remove the joins themselves?

In many production environments, users create views which join many fact and dimension tables.

Developers writing applications query the views instead of querying the tables and writing the joins over and over. Each query that accesses the view selects a different set of columns. When the constraints are defined, the optimizer can limit the tables scanned to the ones that are needed to answer each query.

Join Reordering

When joins can’t be eliminated, the query optimizer uses the Table Constraints to infer information about join cardinalities. The query optimizer may then use this information when performing join reordering.

Let’s look at query 24 of TPC-DS. First, we will run the query without table constraints. In the execution details tab, we can see the elapsed time, the slot time, and the bytes shuffled for the query:

After defining key constraints, we rerun the query and we can see that the improvements in the execution details tab:

These improvements are due to having a better join ordering for the query. The full plan and the join order can be viewed in the execution graph tab. 

User responsibilities

With great power comes great responsibility. Since Key Constraints are not enforced in BigQuery, the user is responsible for maintaining the constraints at all times.

The values of the Primary Key Columns must be unique in all rows and must not have any NULL values.

Each Foreign key must either be null or have a corresponding Primary key row in the referenced table.

If any of these constraints are violated, queries over tables with violated constraints may return incorrect results.

Conclusions

In summary, the BigQuery optimizer can make use of Key Constraints to create better query plans. Such optimizations can lead to major resource savings and faster response times.

To get the benefit of these optimizations, users can define key constraints when creating new tables, and modify their existing tables to add constraints definitions.

Did you know Primary Key Constraints are also used for streaming data from other systems into BigQuery? Check out BigQuery’s Change Data Capture to learn about that functionality.

Sabre is a leading software and technology company that powers the global travel industry. With decades of revolutionary firsts, their team of experts drive innovation and ingenuity across the travel ecosystem. Sabre partners with airlines, hoteliers, agencies, and other travel partners to retail, distribute, and fulfill travel. In this blog post, we’re highlighting Sabre for the DevOps achievements that earned them the ‘Nurturing team culture’ award in the2022 DevOps Awards. If you want to learn more about the winners and how they used DORA metrics and practices to grow their businesses, start here.

As a leader in the travel technology industry, Sabre is constantly innovating and growing, but legacy systems were keeping us from realizing the full benefits of the cloud. Our largest product, Air Offer, is a robust platform that handles millions of flight calculations and shopping requests per second, generating trillions of flight solutions monthly and integrating 7.5 petabytes of data into big data platforms. Air Offer was initially built in the 90s, and while we had shifted it to the cloud, the inherent complexity in the system made it unwieldy and potentially unreliable. This system was too big to fail, but that also meant it was almost too big to change.

The biggest challenge this presented was around scalability. Traffic for services like Air Shopping was doubling every year as shopping algorithms became more complex, as were the computational needs to keep adapting to the changing business dynamics and airline algorithms. Keeping up with these changes was a team effort, but to make lasting improvements, we needed to couple it with better infrastructure, tools, and best practices.

Multi-optimization problem

The company decided we needed a solution that could improve reliability, optimize cost, and increase agility while keeping performance and quality consistent and competitive. Our customers expected quick results when searching for travel deals, and we wanted to improve stability to the point where we could introduce marketing campaigns like flash sales. This would require both cloud mastery and DevOps adoption.

The three main high-level goals the company looked to accomplish with this digital transformation were to:

Improve the velocity and quality of the software and services we provide to our customers

Become more innovative by partnering with Google Cloud from a technology and travel perspective

Improve reliability and security while reducing operational costs 

Solution

To realize these goals, we worked closely with Google Cloud to transform our system and company culture to make better use of the cloud. With the help of Google Cloud, we worked through the core network and security designs to find options that gave our product teams a flexible base to build from. We then looked at specific designs and implementations to match key products like Air Shopping with the Google Cloud services that meet its scalability requirements.

The solution we built involved a mix of public clouds and the data center where we host our applications. This approach took into account the application dependencies and latencies that come from cross-application communications, ensuring a better customer experience. For example, by moving Air Offer from two datacenters that were over provisioned for redundancy to a model where it is spread across four regions using data insights and the flexibility of the cloud to manage capacity and distribution, the company was able to optimize performance and cost in real time while increasing reliability.

Since scale was a major issue going into this digital transformation, we decided the best solution would be to focus on autoscaling. With standard autoscaling, the MIGs are monitored for auto scaling signals in this case CPU utilization to determine when one is experiencing excess demand and start up more servers to handle the increased demand, but this wasn’t a one-size-fits-all solution. For more complicated applications like Air Shopping where servers take several minutes to start up due to data caching requirements, we turned to the predictive autoscaling feature in Compute Engine. This feature keeps extra compute ready to handle periods of high traffic and optimize compute usage without sacrificing customer experience. Solutions like this are invaluable during Black Friday and Cyber Monday when the company must be ready for the highest demand of the year but don’t want to waste money with overprovisioning.

Our company also drove optimization with Compute Engine using spot and preemptible VMs. These gave us the ability to adjust our blend of compute and flexible instances across regions to get the optimal pricing.

The migration of the first workloads took roughly 15 months, starting with Air Shopping. During this time, we learned new ways to work, including implementing a secure CI/CD pipeline and adopting the cloud native concept of Infrastructure-as-Code (IaC) alongside autoscaling and cost management. Once the team was comfortable with the process and started establishing best practices, bringing in other regions to handle compute became much quicker and smoother.

A cross-functional team, including Site Reliability Engineers, platform and software engineers, was established to migrate workloads into Google Cloud. In sticking withWestrum’s definition of good culture – with a shared goal, they removed workflow bottlenecks, addressed blockers directly in standups, and reduced miscommunication. This cooperative approach fostered team pride and efficiency. Ultimately, we built a team of experts across the organization to assist other teams with similar migrations. Success begets success!

Results

This combination of multi-region deployment for high reliability, predictive autoscaling for more efficient resource consumption, and spot and preemptive VMs to optimize cost gave the company the flexibility and continuity we needed to meet customer needs. And, in meeting these needs, we also saw quantifiable improvements to our own business.

Predictive autoscaling specifically has yielded about 10% greater benefit from using basic autoscaling, equaling a projected savings of $3M in 2023 based on projected shipping costs. By knowing how long it takes for a server to start up, the predictive autoscaling logic starts the servers slightly in advance of when they would be needed. This saved the business from having to run around 50% more servers than necessary all day just in case of peak traffic periods without autoscaling, or the extra 10% we needed with basic autoscaling. Additionally, by moving all workloads to Google Cloud with one CI/CD pipeline to deliver changes, Sabre greatly reduced the time to deploy while improving cycle times for new feature releases. 

Before the workload moved to Google Cloud; it took approximately 8-10 hours per release and per location to deploy one of the big applications. We were operating across 4 locations and after adopting a single CI/CD pipeline we have been able to reduce this by 50% to 4 hours per release per region. 

Between the technology and DORA research principles Google Cloud helped us introduce, we saw notable improvements in the five essential characteristics of cloud computing:

Rapid elasticity: To handle spikes in traffic, managed instance groups (MIGs) can scale to several hundred servers in minutes

On-demand self service: Sabre’s teams can reserve capacity and adjust quotas at a project or org level to accommodate changing demand at the click of a button

Broad network access: Google Cloud simplifies infrastructure by giving teams easy access to the computing resources, reports, and alerting across a variety of devices

Measured service: With Google Cloud operations and other tools, Sabre can meter and scale resources to optimize user experience while tracking cost

Resource pooling: By using cloud-native capabilities like autoscaling and separating resource pools by client type, Sabre gives customers the ability to scale and use resources within their contracted volumes without human intervention

Stay tuned for the rest of the series highlighting the DevOps Award Winners and read the 2022 State of DevOps report to dive deeper into the DORA research.

In this blog post, we’re highlighting VMO2 for the DevOps achievements that earned them the ‘Continuous improvement with DORA’ award in the2022 DevOps Awards. Launched in June 2021, Virgin Media O2 (VMO2) combined the UK’s largest, most reliable mobile network with a broadband network that offered the fastest widely-available broadband speeds. Since then, they’ve accrued 47.7 million UK connections across broadband, mobile, TV, and home phone, all in the name of their mission of upgrading the UK. To learn more about the 2022 DevOps Awards winners and how they used DORA metrics and practices to grow their businesses, start here.

As a previous DORA DevOps Award winner, we deeply understand how DevOps creates a consistent, productive work environment. Since receiving the award we’ve spread what we’ve learned to teams across the company, and have even started giving presentations at conferences and written blog posts showcasing our DevOps evolution to a wider audience.

Even with all of this transformation, we realized there was always more to learn and grow, especially with the complications of a changing telco landscape. Not only did we still have highly coupled data dependencies on legacy infrastructure and multiple, overlapping processes that required heavy outsourcing, we still had issues born from merging two huge telco operators. After reviewing our systems, we realized that we were running servers and clusters from most of the major providers.

Objectives

To overcome all of this technical debt, the company realized that we needed to migrate the most critical of these source systems to Google Cloud on the VM side while also improving the underlying tech and culture. Specifically, our goals were to:

Remove complexity from the overall organization

Provide personalization and support to customers

Create future-ready infrastructure in the most cost-effective way

Solution

To accomplish this, we partnered closely with Google Cloud to create a strategy that centers around three key pillars: DevOps, MLOps, and Analytics Engineering.

For our DevOps decentralization effort, we shifted from a centralized model where an individual team was responsible for all building efforts, to a decentralized one, with 28 teams building together daily, including over 200+ engineers and analysts. Each of these teams have their own data products with mostly independent code bases or data models, but all of them share common libraries, best practices, and principles. And as these teams discover new best practices, they are shared across teams, further driving a culture of growth and collaboration.

Since the beginning of our MLOps journey in early 2022, we now have over 50 data scientists and engineers, with company-wide adoption of Vertex AI pipeline-based in-house platform, leading to reduced time-to-production and increased scalability.

Time-to-market: By using CDC-tracked raw data with BigQuery and our adopted dbt (a data modeling tool) cloud, an 8-person team helps 160 analysts use self-service tools to make more informed moves without waiting for data.

Data ingestion: To streamline ingestion, we instituted a Vortex platform to ingest multiple forms of third-party data for batch and streaming, while we collaborate with other teams to ingest internal data sources using Ab Initio and Cloud Composer.

Feature rollout: With a repository of ML templates and MLOps best practices and a shared MLOps Python package, our teams have scalable solutions to deploy quickly and securely following CI/CD practices.

The Analytics Engineering team uses dbt to continue their mission of ensuring that data is trusted and accessible so that teams can develop at pace in a controlled environment to get more value from data. Thanks to this and our other efforts, the teams production area processes 250TB daily with 60k queries run against created data models and contain 91k lines of dbt code.

Results

This commitment to continuous improvement has yielded tangible results in terms of value creation through cost reduction, revenue generation, and customer satisfaction. In the short term, this includes:

Driving digital store conversions (CVR) of 458% as well as tripling the average revenue uplift against the legacy through commercial and recommendation engines, and an additional 6% CVR increase from recommending relevant products

Saving network engineers 3000 days of working time, which comes out to £800k cost savings

Creating 50% of the third-party incumbent value for 10% of the annual spend through our internal call routing

In addition to these immediate returns, we have also seen medium-to-long-term returns that will help unlock the company’s future growth, including:

Increasing first-time resolution in the channel that the customer chooses through omnichannel, data product-driven solutions, and putting their digital channel share on the road to reaching 80% by 2026

Helping the company make better decisions when it comes to investing in the network — part of a £4B program to upgrade the UK

Bringing mobile and fixed services together in converged experiences for Virgin Media O2 customers, saving millions of pounds through decommissioning, work efficiencies, and shared practices and techniques

Stay tuned for the rest of the series highlighting the 2022 DevOps Award Winners and read the 2022 State of DevOps report to dive deeper into the DORA research.

※この投稿は米国時間 2023 年 7 月 8 日に、Google Cloud blog に投稿されたものの抄訳です。

このたび、Google Cloud の単一プロジェクトにおける複数 Firestore データベースのサポートを開始しましたので、お知らせいたします。1 つのプロジェクトに複数のデータベースを作成し、顧客データやマイクロサービス、開発 / テスト / ステージングの各種環境を隔離して管理できるようになりました。

この複数データベースのサポートには以下の機能が含まれます。

Firestore データベースの CRUD 管理: Firestore で新しい API エンドポイント、Terraform リソース、gcloud CLI コマンド、Firebase CLI コマンドを使用して、1 つ以上の Firestore データベースを管理できるようになりました。

Identity and Access Management による条件付きアクセス制御: IAM Conditions または Firebase セキュリティ ルールを使用して、異なる Firestore データベースに異なるセキュリティ ポリシーを適用できるようになりました。

Firestore データベースの 2 つのモードをサポート: 同一プロジェクト内で新しいデータベースを Firestore の Native モードと Datastore モードのどちらでも作成できます。

すべての Firestore リージョンをサポート: 同一プロジェクト内で新しいデータベースを、サポートされているすべての Firestore リージョンに作成できます。

課金処理: データベースの費用を Cloud Billing と BigQuery を通じて個別にトラッキングできます。これによって、各データベースの使用に対する消費量の確認と予算の編成が容易になります。

Cloud Monitoring: Firestore の Cloud Monitoring 指標と統計情報がデータベース レベルで集約されました。

サンプルのチュートリアル

Firestore はお客様のアプリケーションでも高い利便性を発揮できるかもしれません。エンドツーエンドの例で、その仕組みご確認ください。

Firestore データベースを作成するには、データベースの識別子、モード、ロケーションを指定する必要があります。Firestore データベースには複数の作成方法があります。たとえば、gcloud CLI を使用して Firestore ネイティブ データベースを「test」という名前で us-west1 ロケーションに作成する場合は、以下のコマンドを使用します。

データベースが作成されたら、Firestore / Datastore コンソールからアクセスできます。

Firestore データベースが不要になったら、次のコマンドを実行して削除できます。

これを実行するだけで、あとは Firestore が自動的にデータの削除を完了します。

作成したデータベースで Firestore クライアント ライブラリを使用する場合は、適切な Firestore データベース インスタンスを取得できるようにデータベース名を指定します。データベース名を指定しない場合、クライアント ライブラリは元の（デフォルトの）データベースに基づきます。Java 対応の Firebase Admin SDK を使用して「test」という名前のデータベースを記述する例は、次のとおりです。

他の言語を使用した例については、データベースへのアクセスをご覧ください。

次のステップ

Firestore での複数データベースの設定および構成方法について詳しくは、こちらのドキュメントをご覧ください。

– Firestore、ソフトウェア エンジニア Sichen Liu
– Firestore、シニア プロダクト マネージャー リード Minh Nguyen

Large Machine Learning (ML) models – such as large language models, generative AI, and vision models – are dramatically increasing the number of trainable parameters and are achieving state-of-the-art results. Increasing the number of parameters results in the model being too large to fit on a single VM instance thus demands distributed compute to spread the model across multiple nodes. Google Kubernetes Engine (GKE) has built-in support for NCCL Fast Socket, to help improve the time to train large ML models with distributed, multi-node clusters.

Enterprises are looking for faster and cheaper performance to train their ML models. With distributed training, communicating gradients across nodes is a performance bottleneck. Optimizing inter-node latency is critical to reduce training time and costs. Distributed training uses collective communication as a transport layer over the network between the multiple hosts. Collective communication primitives such as all-gather, all-reduce, broadcast, reduce, reduce-scatter, and point-to-point send and receive are used in distributed training in Machine Learning. 

The NVIDIA Collective Communication Library (NCCL) is commonly used by popular ML frameworks such as TensorFlow and PyTorch. It is a highly optimized implementation for high bandwidth and low latency between NVIDIA GPUs. Google developed a proprietary version of NCCL called NCCL Fast Socket to optimize performance for deep learning on Google Cloud.

NCCL Fast Socket uses a number of techniques to achieve better and more consistent NCCL performance.

Use of multiple network flows to attain maximum throughput. NCCL Fast Socket introduces additional optimizations over NCCL’s built-in multi-stream support, including better overlapping of multiple communication requests.

Dynamic load balancing of multiple network flows. NCCL can adapt to changing network and host conditions. With this optimization, straggler network flows will not significantly slow down the entire NCCL collective operation.

Integration with Google Cloud’s Andromeda virtual network stack.This increases overall network throughput by avoiding contentions in virtual machines (VMs).

We tested (NVIDIA NCCL tests) the performance of NCCL Fast Socket vs NCCL on various machine shapes with 2 node GKE clusters.

The following chart shows the results. For each machine shape, the NCCL performance without Fast Socket is normalized to 1. In each case, using NCCL Fast Socket demonstrated increased performance in a range of 1.3 to 2.6 times faster internetwork communication speed.

As a built-in feature, GKE users can take advantage of NCCL Fast Socket without changing or recompiling their applications, ML frameworks (such as TensorFlow or PyTorch), or even the NCCL library itself. To start using NCCL Fast Socket, create a node pool that uses the plugin with the –enable-fast-socket and –enable-gvnic flags. You can also update an existing node pool using gcloud container node-pools update.

To achieve better network throughput with NCCL, Google Virtual NICs (gVNICs) must be enabled when creating VM instances. For detailed instructions on how to use gVNICs, please refer to the gVNIC guide. 

To verify that NCCL Fast Socket has been enabled, view the kube-system pods:

And the output should b similar to:

To learn more visit GKE NCCL Fast Socket documentation. We look forward to hearing how NCCL Fast Socket improves your ML Training experience on GKE.

IT infrastructure is essential to the success of any organization. Unfortunately burnout is a very real threat to many IT teams’ wellbeing and productivity, compromising their ability to deliver and maintain that infrastructure. We’ve seen that utilizing the public cloud can reduce burnout, but many IT teams still aren’t sure how to make that switch. That’s why we published a new ebook where we follow a day in the life of On-Prem Pete and Cloud Carrie, two infrastructure managers whose workdays play out very differently. By following their paths, it quickly becomes clear how switching to cloud can make life much easier for both IT staff and developers.

As you may know from personal experience, IT and infrastructure teams are often under a lot of stress, some of which might be exacerbated by non-ideal IT landscapes in on-prem data centers. A few of the most common reasons that IT teams may feel stressed include the ever-changing nature of technology, the push for integrating new AI solutions, the need for staff to be constantly available, the responsibility to keep data safe, the pressure to perform, and work-life imbalance. 

All of these pressures can take a toll on IT teams, sometimes leading to burnout — a state of physical and emotional exhaustion that can cause a loss of motivation or efficiency. In fact, as you can see in the infographic below, 74% of employees have experienced burnout at some point in their careers, and there was a 500% increase in searches for ‘pandemic burnout symptoms’ from 2020 to 2022. Clearly, burnout is a very real issue.

Making this even more challenging? A lot of people don’t even know when they’re burned out — especially since burnout symptoms can vary from person to person. However, there are some common symptoms that you should look out for: 

Exhaustion. This is one of the most common symptoms of burnout. You feel tired all the time, even when you’ve had a good night’s sleep. You may also feel like you are running on empty.

Cynicism. With this common symptom, you may start to feel like nothing you do makes a difference, or that your work is pointless.

Lost motivation. You may start to feel like you don’t care about your job anymore, or that you’re just going through the motions.

What’s worse is that in addition to the human impact, burnout can also impact the business. That’s why businesses should take avoiding burnout among employees seriously! Thankfully, there are many things you can do to prevent burnout, with some of the most important being:

Setting boundaries. It’s important to establish boundaries between your work life and your personal life. This means not checking work emails or messages outside of work hours. It also means taking time for yourself, even when you are busy, and having clear conversations with your management teams about what is and isn’t realistic. 

Taking breaks throughout the day. This will help you from feeling overwhelmed. You also need to take vacations and use your sick days.

Talking to someone. It’s important to talk to someone about how you are feeling. This could be a friend, family member, therapist, or counselor. Talking to someone can help you feel better and get the support you need.

Making changes. If you are feeling burned out, it may be time to make some changes. This could include changing your job, your work environment, or your lifestyle.

Understanding burnout — especially its symptoms — is the first step you can take to avoiding it. The second step you can take is finding ways to accomplish your work goals as optimally and efficiently as possible. We hope our free ebook and infographic can help you accomplish all of your IT goals — while maintaining your wellbeing and productivity along the way.

According to this year’s State of FinOps report, across cloud providers, organizations employ 4.1 cloud cost-management tools. Four point one! We’ve observed a similar trend with our own customers. On average, they report using four tools — native, third-party, and homegrown — to make sense of their cloud bills. Astoundingly, we’ve learned from some of larger enterprise customers that they may have 10 or more cost-management and operation tools. When companies get in this situation, making decisions is a challenge.

The industry is taking note with efforts to standardize open billing data. And organizations like yours are going to great lengths to demystify their cloud spend. But when it comes to multicloud reporting, enterprise customers still often face the age-old question : Buy or build? 

To illuminate the topic, we talked to one enterprise customer, UKG, which provides human capital management (HCM) and workforce management solutions. Recently, their Director of Cloud FinOps, Peter Crenshaw, joined us on the Framing Up FinOps podcast to share his thoughts about UKG’s Cloud FinOps journey and in-house cost-management tooling. 

Building an advanced reporting solution

About three years ago, when leaders at UKG decided they needed to clarify the organization’s cloud costs, they approached Crenshaw to begin reporting on Google Cloud spend. “We had a massive product deployed but didn’t understand the bills,” Peter explains. “Eighteen months or so ago, we ran into issues finding a third party that could process and manipulate data based on our reseller discounting, then present it in a way that would work.” 

Like many enterprises, UKG has a large cloud footprint. They had multiple and disparate advisees, advisors, providers, and tools, each producing data that didn’t match up. Together with teammates, Peter started a small FinOps practice. In his words, they began with “rudimentary Excel and menial efforts.” They needed to decide whether to build their own cost-management solution or to go with a third-party vendor. Ultimately, they partnered with Google Cloud — first, to find a solution to view current costs, then to expand on its use as their team and FinOps practice matured. 

Since then, the official team has grown from one to four. And they’ve consolidated their cost-management tooling to align with industry averages. To better understand and report on UKG’s cloud spend, Peter and his team worked with Google Cloud to build an advanced reporting solution. 

Managing cloud cost reporting with Looker and BigQuery

For cost reconciliation and granular visibility into cloud cost breakdowns, Peter and team elected to build their solution on Looker, a platform designed for insight-powered workflows and applications. Additionally they leveraged Cloud Billing data to BigQuery for advanced billing data at the resource level and the Google Cloud billing console for validation. 

In assembling UKG’s FinOps team, Peter departed from industry recommendations to establish a cross-functional team with engineers and cloud financial analysts. Instead — leveraging the strengths of his organization — his team partners with the in-house experts of their SRE teams. “We looked at FinOps as an analytics team. We’re not engineers, and we don’t build tooling or software. . . We know and can show the data and partner with teams that have those skill sets to make strides and advance,” he explains. 

PO processing time drops from weeks to a day 

When we asked Peter about the benefits and outcomes of building tooling on Looker, he underscored the depth of visibility, the ability to alert teams to real-time data, and increased invoicing efficiency. Currently, he and the FinOps team have been evangelizing the advantages of labeling. With real-time reports and demos, they’re showing internal teams the true costs of running various components, services, or products. These data and demos have elicited a measure of FinOps buy-in across the organization and have begun to precipitate a culture shift.

Additionally, UKG’s Finance and FinOps teams are seeing the biggest wins in the reduction of time spent on tasks. PO invoice processing at UKG used to take as many as 2 weeks in the payment cycle for approvals. But with advanced reporting and custom templates, Peter and his team can process their own data within 30 minutes of receiving an invoice. Organization wide, UKG has reduced PO invoice processing time from two weeks to one day. 

As Peter notes, and as we’ve shared here before, getting visibility into cost drivers is oftentimes the first and most powerful step in the cloud cost-optimization journey. “The data and visibility has been [most] impactful with my team. Before, we were drowning in spreadsheets and watching Excel crash every day,” he recalls. “Our reporting is much easier now with the same inputs, same reports, and same visibility. But also, the entire org is benefiting as we’re peeling back the layers, seeing what’s under the hood. [We’re] able to tell people what they have, what their costs are.”

Transitioning to a value-stream model

UKG is transitioning to a value-stream operational model, where a product owner is  responsible for their own cloud budget. Previously, they had one cost center across the entire cloud bill, then manually completed journal entry to assign cloud costs. With the new system, which will eventually incorporate labeling and chargebacks directly into POs, individual teams will be responsible for their own cloud costs. 

But labeling remains an organizational challenge. As such, the challenge to the FinOps team will be to clearly articulate costs, break them down month by month, and not only enter them but also assign them in the payment system. “The granularity of the reporting isn’t as it could be, but we’re getting there,” Peter says.

Build, buy, or both?

Whether your organization is a well-established enterprise or a small business just getting started with a Cloud FinOps practice, determining whether to build, buy, or both is really about understanding your business objectives, then selecting the right tools to meet them. 

There are plenty of out-of-the-box Google Cloud billing tools to make sense of your cloud spend. But if, like UKG, your enterprise is considering or working on building an advanced reporting solution, we recommend a rationalization exercise to understand what you have as a first step. 

To optimize reporting tools, we’ve found that organizations must often work backward to understand their desired outcomes and business objectives. Some critical questions to consider include:

What are the goals of your FinOps org? 

What are you trying to accomplish in the near and long term?

Where is your team’s time best spent? 

Do you have the appropriate tool set to augment your time and accomplish your goals? 

Hopefully, you’ve taken away some useful insights to help you get more value from your cloud investment. If you need support to determine the right solutions for your company, we’re happy to help. There’s much more to learn, so please remember to join us every other week for our newest episode of Framing up FinOps on Twitter Spaces.

Special thanks to Googlers Sheri Cunningham, Cloud FinOps Team Lead, and Nick Davidson, Technical Account Manager for UKG, as guest speakers on our Framing up FinOps.

Here at Google Cloud we are always trying to find new ways to simplify how our customers troubleshoot. We’re excited to announce the introduction of a new troubleshooting experience: recommended interactive playbooks for Google Kubernetes Engine (GKE).

When dealing with issues that may be new to you, but that we’ve seen commonly in the past, these new playbooks can help you more quickly resolve issues and improve your Mean Time to Resolution, or MTTR. 

Let’s take a quick look at one of these new example playbooks.

Let’s say we have a GKE cluster and an application requesting more resources than are available to it, such as memory or CPU. In that situation, it’s often the case that a Pod (or Pods) will be marked as ‘unschedulable’.

A Pod being marked as ‘unschedulable’ is a common issue and something we have documentedextensively, but let’s see how we can simplify the troubleshooting process.

In the screenshot below we’ve highlighted the notification from the cluster view that Pods are unschedulable.

If we click this notification we see a screen appear offering us a few ways to better understand this issue:

Clicking into the playbook, we can see a lot of information relevant to the issue at hand including relevant logs, metrics, and suggested next steps:

We can see from the logs and metrics that the Pods of the Deployment have requested more memory than is available, but that the node has ample resources available and there are no maximum limits on Pods being set. So to resolve this issue, we’ll need to modify the amount of memory the Pod requests, or increase the size of our cluster. 

This dashboard is also customizable, so if you’d like you can add or remove components based on what’s most pertinent to you and your organization.

Finally, at the bottom of the playbook, under ‘Future Mitigation Tips’, you can also create an alert policy to look specifically for this issue:

When this alert fires, you’ll be able to acknowledge the incident or click the policy link to jump straight into this dashboard and begin troubleshooting:

This week we’re making two playbooks available: Unschedulable Pods, and a playbook for troubleshooting repeated attempts of a deployment crashing, commonly known as CrashLoopBackOff. We have playbooks for Memory and CPU scaling issues coming soon.

Both will appear as notifications for clusters where issues are present, and we hope this helps you in your troubleshooting journey! As always, if you have any questions or feedback on the product, please let us know by leaving feedback under the question mark icon of the page.

The race to increase global mobile broadband access has been on for years. As more people depend daily on fast, reliable internet, telecom companies are looking to continuously improve and expand their services. The challenge for many is: how to accomplish this on a massive scale quickly and cost effectively?

The answer lies in advanced data analytics.

MTN Group is the leading telecom provider in Africa, offering voice, data, fintech, and enterprise solutions to over 270 million customers across 19 nations. It was one of the first companies on the continent to build a data science platform that would de-silo information across all markets.

Let’s look at how MTN Group uses Google Cloud solutions and partners with Deloitte to achieve its goals and maximize the impacts.

A first for African telecoms

MTN Group launched its Advanced Data Analytics Management (ADAM) Platform to transform how it gathers, analyzes, and uses data across the business.

“The ADAM Platform is complex, but its objective is simple: improve customer experiences by understanding their preferences in near real-time and tailoring experiences accordingly,” says Mohanoe Mokhitli, General Manager, Data and Analytics, Group Business Intelligence Competency Centre, at MTN Group. “Google Cloud was the clear choice as the technology foundation due to its connectivity across Africa, scalability, and analytics tools.”

The company chose to work with partner Deloitte to build a blueprint of the platform on Google Cloud that could be deployed in all 19 markets it operates in. Deloitte provided expertise in Google Cloud solutions and a presence across many African nations to help guide the project from its inception.

“Deloitte brought the skills to make this project work at scale, helping us to cover everything from change management and decision-making to the actual platform tooling,” says Mokhitli. “The partnership just makes sense. We’ve benefited from a lot of knowledge transfer thanks to how closely we’ve worked with Deloitte on the ADAM Platform.”   

Deloitte brought essential expertise as MTN looked to adopt a new technology foundation. An important focus was on ensuring the platform was built efficiently and could scale to handle the enormous data quantities associated with 270 million subscribers.

Tapping into the power of AI/ML

The ADAM Platform relies on BigQuery as its central data warehouse, as well as several other Google Cloud analytics technologies such as Dataproc to accelerate the process of capturing data and gaining insights. Vertex AI has been crucial to the platform’s success, helping make sense of billions of records quickly and efficiently.

With the ADAM Platform up and running, MTN is now processing roughly 4 trillion data records monthly and democratizing insights to derive palpable business value from its data.

“In addition to the sheer power of Vertex AI and BigQuery, we have also benefited from its ease of use,” says Dr Quentin Williams, Associate Director at Deloitte Consulting. “With Google Cloud, we can make sure MTN business users can take advantage of the technologies regardless of their IT experience. This has helped to drive adoption among our business units, which translates to a greater customer impact.”

Furthermore, ADAM enables MTN Group to gain richer, more timely insights into customer needs. For example, sales teams can use insights from the ADAM Platform to decide which services should be packaged together in offers to certain customers.

With access to the platform, data-driven decision making is now more prevalent and streamlined across the MTN Group, as previously siloed initiatives are connected to increase visibility across all markets. Teams benefit because there is less redundant work and they can take quicker action on what’s working well. The result is that each business unit can contribute more to building customer satisfaction and loyalty and reducing churn.

Deloitte and Google Cloud have helped MTN Group manage the technical demands of the project so that MTN Group employees can focus on educating users and increasing adoption.

“If you look back at our business when this project kicked off and compare it to where we are now, we have closed a significant skills gap that held us back from data-driven decision making,” says Mokhitli. “With help from Deloitte and Google Cloud, the ADAM Platform increases our ability to build our business on data-driven insights, which ultimately provides our customers with even better experiences.”

Read more customer stories to learn how Google Cloud and partners like Deloitte can help to transform your business.

Cloud SQL is Google Cloud’s enterprise-ready, fully managed database service for running MySQL, PostgreSQL and SQL Server workloads. It’s used across industries from digital services to banking to retail, with more than 95% of Google Cloud’s top 100 customers using Cloud SQL today. As more demanding workloads move to the cloud, some of you have been asking us for higher performance and availability. In addition, you’ve been asking us for more choice and flexibility to optimize your configurations for your individual workload needs.

Today, we’re announcing the Cloud SQL Enterprise Plus edition for MySQL and PostgreSQL, featuring three major enhancements. First, we have significantly accelerated both read and write performance with a variety of software optimizations, improved machine types and configurations, and an integrated SSD–backed data cache option. With Cloud SQL Enterprise Plus for MySQL, we deliver up to 3x higher read throughput and up to a 2x improvement in write latency compared to the current Cloud SQL offering. Second, we deliver near-zero downtime for planned maintenance operations and a 99.99% availability SLA inclusive of maintenance to satisfy your availability goals for your business-critical applications. Third, we enhanced the data protection capabilities with 35 days of log retention, a must-have for all organizations with strict compliance requirements. Cloud SQL Enterprise Plus edition addresses these needs while building on the already proven capabilities of Cloud SQL. 

The existing version of Cloud SQL will continue with no changes to features or pricing, but will now be known as the Cloud SQL Enterprise edition. 

Exploring the innovations in Enterprise Plus for MySQL

Cloud SQL Enterprise Plus for MySQL delivers superior performance through co-optimized hardware and software configurations while maintaining full open-source compatibility. For read-intensive transactional workloads, Enterprise Plus edition for MySQL, with configurable data cache, delivers up to 3x higher read throughput compared to Enterprise edition. Data cache leverages flash memory as a way to transparently extend caches based on DRAM, lowering read latency, improving throughput and scaling to larger data sets. Additionally, software optimizations deliver up to a 2x reduction in transaction commit latency and up to a 2x improvement in write throughput, making Cloud SQL the best destination for your most demanding MySQL workloads.

“We are delighted to join forces with Google Cloud in launching Cloud SQL Enterprise Plus edition for MySQL,” says Jonathan Blais, CTO and Co-founder of Everflow Technologies. “This new edition has enabled us to immediately reap the benefits of enhanced performance, availability, and data protection without making any modifications to our application. In particular, we have observed more than 3x improvement in query performance for clients who run intensive reports on our platform after incorporating the Data Cache. We look forward to partnering with Google Cloud and Cloud SQL on future innovations.” 

Availability and reliability are a key focus and Cloud SQL Enterprise Plus edition offers a 99.99% SLA, inclusive of maintenance. During planned maintenance, Enterprise Plus instances use rolling updates behind the scenes and additional hardware resources to reduce downtime to less than 10 seconds.

Enterprise Plus edition also provides a higher level of data protection, with 35 days of log retention for point-in-time-recovery, supporting demanding regulatory needs and data protection requirements that your business critical workloads need. 

“At Workday, it’s important to be able to offer great user experience to our customers and as we moved to the public cloud, it was clear that we needed a high-performance database to support our requirements. We worked closely with the Google Cloud team to invest in Cloud SQL improvements and with the launch of the Cloud SQL Enterprise Plus edition, we now have options for high performance and availability for our most demanding Workday workloads, leveraging the higher read and write performance, higher availability, and minimal planned downtime. We are really excited to co-innovate with Google and benefit from these new capabilities along with Enterprise-grade features of the managed services,” says Chirag Andani, VP, Persistence and LifeCycle Engineering at Workday.

Enterprise Plus for PostgreSQL

Cloud SQL Enterprise Plus for PostgreSQL delivers up to 2x improved write throughput compared to Enterprise edition. Similar to the MySQL offering, Cloud SQL Enterprise Plus for PostgreSQL co-optimizes hardware and software to deliver this improved performance. 

All of the availability, reliability and data protection enhancements mentioned for MySQL, are also available with Cloud SQL Enterprise Plus for PostgreSQL: a 99.99% availability SLA inclusive of maintenance, reduced planned downtime of less than 10 seconds, and a higher level of data protection with 35 days of log retention. 

“Partnering with Google Cloud and using Cloud SQL Enterprise Plus edition for PostgreSQL has allowed us to immediately take advantage of increased performance, availability and data protection while maintaining the open-source PostgreSQL compatibility that our applications require. With our very first workload, we saw 2x improvement in sustained performance (TPS and query latency) out of the box,” says Richard Helms, Manager, Database Engineering, Salesloft.

Getting started

Cloud SQL Enterprise Plus edition is now available in select Google Cloud regions via gcloud and API and will be available in the cloud console by July 17, 2023. Cloud SQL Enterprise Plus edition for SQL Server will follow later.

Getting started is easy: visit the Cloud SQL console and create a database instance with just a few clicks. New Google Cloud customers can create a free trial instance and get $300 in free credits. To learn more about Enterprise Plus edition, see our documentation.

As companies modernize their applications and infrastructure, many IT stakeholders are looking to open source technologies to help them do so. One area where open source is gaining traction is in relational database management systems (RDBMS). Open-source RDBMSs such as PostgreSQL are becoming increasingly popular alternatives to commercial RDBMSs such as Oracle and Microsoft SQL Server. There are several reasons for this shift. 

First, open-source RDBMSs are often more cost-effective than legacy, proprietary RDBMSs. They are typically free to download and use, and there are no licensing fees. 

Second, open-source RDBMSs are often more flexible and scalable than legacy RDBMSs. They can be easily customized to meet the specific needs of an organization, and they can be scaled up or down as needed. 

Due to these factors, open source RDBMSs are becoming a popular choice for organizations of all sizes. They offer a number of advantages over legacy RDBMSs, including cost-effectiveness, flexibility and scalability.

Migrating from Oracle/SQL Server to PostgreSQL

Migrating from Oracle/SQL Server to PostgreSQL can be a challenging task. The migration strategy will vary depending on your specific goals, but there are some general strategies that can help to smooth the migration process: 

Utilize Database Migration Assessment (DMA): There are a number of assessment tools that will provide insights detailing the level of effort required to modernize an Oracle or SQL Server database to PostgreSQL. To expedite this step, customers can leverage the Google Cloud first-party tooling called DMA to assess Oracle/SQL Server database complexity. It’s Google Cloud’s “no cost” database assessment solution, which estimates the migration effort, cost, and anticipated return on investment. We’ll talk about DMA in more depth below. 

Leverage code/schema conversion tooling: These tools can help to reduce the risk of human error and improve the overall efficiency of the migration. Leverage  Google Cloud tooling to automatically convert your schema and code from Oracle/SQL Server to PostgreSQL.

Migrate data in stages: Migrating all of your data at once can be a risky proposition. It is often better to migrate your data in stages, starting with a small subset of data and then gradually migrating more data over time. Depending on your requirements, we can help you build a data migration strategy. Whether you can tolerate downtime or need real-time replication, we offer data migration solutions to meet your requirements.

Make application changes: Switching a database engine often involves application changes. We can help scan your application repositories and identify the SQL statements that may need to be altered to run on the new database engine.

Test the migrated data: Once you’ve migrated your data, it’s important to test it to make sure that it is working properly. There are a number of tasks required in this stage: First you need to validate that all the data is synced between the source and the target; you can leverage our tooling for this task. Second, you’ll need to perform application functional testing to validate that the application behaves as expected. Lastly, we recommend performance testing.

By following these general strategies, you can help to ensure that your database migration from Oracle/SQL Server to PostgreSQL is smooth and successful. 

The rest of this blog will focus on our database assessment process, which is the starting point in this cloud journey. A database assessment brings together industry best practices, database experts and data collection tooling to provide you with a migration strategy to tackle your Oracle and SQL Server migrations to PostgreSQL. 

Migrating to PostgreSQL: Start with a Database Migration Assessment

The Database Migration Assessment (DMA) is a no-cost customer engagement in which we collect database metadata and deliver a detailed customized customer readout. The collected data contains detailed metadata spanning database objects types, PL/SQL or T-SQL code, database features in use, current resource usage and the workload characteristics. This data is collected using our *new* first partyDMA tool.

Once the metadata is collected, it gets processed and our database expert team delivers a detailed customized readout containing:

Google Cloud target state database recommendations based on your current on-premises database workload characteristics.

Migration effort: This reflects the level of effort required in hours to convert the Oracle/SQL Server schema and code to an open source database like PostgreSQL.

Right-sizing for the cloud: Current on-premises resources usage information is used to right-size your Google Cloud database, so that you only pay for resources that you actually need.

Complete migration plan: Identify your first-mover databases and create migration waves for subsequent databases.

Below is an excerpt of a customer readout, in which we looked at 14 Oracle databases. In this example, our customer wanted to modernize from Oracle to either Cloud SQL for PostgreSQL or AlloyDB for PostgreSQL. Originally, the customer’s perception was that their Oracle databases would be too complex to modernize and that it would require too much effort. Once we ran the migration assessment with DMA, we were able to determine that 89% of their database environment could be automatically converted by using Google Cloud schema/code conversion tooling. We determined this based on the graph below, in which each database is listed and the compatibility between Oracle/SQL Server and PostgreSQL is displayed. 

For each database, we categorize database objects and code into three buckets. The green bucket identifies the database object that can be automatically converted using the Google Cloud tooling. The yellow bucket identifies the database objects/code that require some manual work. Lastly, the red bucket identifies the database objects that need to be completely refactored because the Google Cloud schema/code conversion tooling cannot automatically convert these objects. In summary, the more green there is on the chart, the more objects we can automatically convert using Google Cloud conversion tooling, and the less effort is required by the customer.

For this customer, the results were better than they expected. Originally, their overall impression was that there was a lot of PL/SQL code in these Oracle databases which were making these environments sticky. However, this visualization showed that most of their Oracle objects and code could be automatically converted.

Next, we showed the customer the level of effort required to convert each of their databases. The graph below displays all the databases ordered by the level of effort (shown in hours of time) required to complete a migration. The first database listed will take the least amount of time to migrate, whereas the last database will take the most amount of time to migrate. This data is useful to help you identify the first-mover databases and define your migration waves. In other words, you can define a database group to migrate in phases.

So, how do you start? As a recommendation, we suggest starting with the least complex database first as opposed to the most complex. This will allow you to begin building a migration blueprint with repeatable processes and get you comfortable with the Google Cloud schema/code conversion and data movement tooling. These learnings in combination with a repeatable process will set the stage to expedite migrating your more complex databases. 

This customer was delighted with these results and concluded that the level of effort required to modernize their Oracle databases to AlloyDB offered a high return-on-investment (ROI), as they were moving away from an expensive licensing/support structure to a cost-effective fully managed database service. 

In this example, we only provided an excerpt of the customer readout that we delivered to the customer. However, a full customer readout provides a wealth of information. We can drill down multiple levels, right down to the individual database objects that make up the hours of effort calculation.

Key features

The DMA tool is open source and is available on Github. It currently supports Oracle and SQL Server databases:

For Oracle,  DMA supports both AWR and non-AWR (statspack) database assessments based on customer licensing policy. 

For MS SQL, DMA leverages Perfmon data and only collects metadata.

The table below lists the type of data that is collected with the DMA tool.

Conclusion

If you are currently using Oracle or SQL Server databases and are looking to modernize them onto an open-source database like PostgreSQL,  you can download the DMA tool from our Github repository. Or, reach out to your Google Cloud sales representative and schedule an assessment. Our database experts can provide recommendations, best practices and guidance on modernizing these databases, which in turn will provide a data-driven approach to assessing if modernizing your Oracle or SQL Server database is right for you.

Kubernetes is an increasingly key part of the application deployment strategies at large organizations, and one of the most recommended options for the teams we work with. An idea brought to life at Google, organizations throughout the world use containers and Kubernetes on-premises, in Google Cloud, or in a multi-cloud scenario, and it has emerged as a leading application deployment platform. And because it’s open source, anyone can pop the hood, so to speak, to see how each component of it works, creating a trusted, verifiable framework that users can rely on.

Customers begin their Kubernetes and containerization journeys using various offerings in our product portfolio. Some, like Colgate, started their modernization journey with Kubernetes. Colgate is an $18B global consumer products company with ~34,000 diverse and dedicated people serving over 200 countries and territories. Through science-led innovation, they drive growth and reimagine a healthier future for all people, their pets, and our planet. 

During ideation, they talked through various considerations: What technology strengths does their organization have? What skills do their teams need? What will this initiative look like a decade from now? They implemented the following Kubernetes-focused architecture:

Supporting a variety of use cases and teams

Google Cloud also helped Colgate break new ground over the years, especially in the areas of cloud-native networking, security, monitoring, pub/sub, managed containerization, and multi-tenant environments.

Over time Colgate began to leverage Google’s managed container portfolio, which includes Cloud Run. Cloud Run lets you run containers on top of a serverless platform, unlocking workload possibilities for public websites, private services, APIs and batch jobs and eliminating a lot of the time spent on infrastructure management. Cloud Run also requires no prior knowledge of Kubernetes or containers.

Simplified enterprise-level management

Many teams have found that they prefer the serverless, hands-off approach that Cloud Run provides, and Colgate now evaluates Google Cloud’s serverless solutions alongside GKE for any applications destined for the cloud. At the same time, Cloud Run lets them continue to leverage their investment in workloads based on the Open Container Initiative. 

For example, Cloud Run is designed for Kubernetes compatibility with consistent management capabilities such as the ability to manage resources using kubectl via the config controller, and the ability to browse logs and metrics from both platforms in Cloud Logging and Monitoring. Cloud Run and GKE data planes are also interoperable, allowing Cloud Run and GKE services to be exposed behind a VPC behind private IPs using an internal load balancer. Cloud Run as an option has contributed to faster innovation, allowing Colgate to bring smiles to many more faces globally.

Google’s managed container offerings provide a composable and comprehensive set of solutions for customers’ applications. At Colgate, container-based managed services are used across the stack: on the front-end, where they use Identity-Aware Proxy to manage authentication and External Load Balancers to handle incoming traffic with high availability and low latency; at the application layer, where they can choose from Cloud Functions, Cloud Run, or GKE, depending on the level of control they need over the application; and at the internal load balancing level where NGINX® controllers serve internal applications. Together, these managed services ensure that Colgate has the flexibility to choose the right toolchain and maximize their goals for each use case. 

Colgate wanted to build internal applications on Cloud Run in a way that complied with their organization’s policies while maximizing developer productivity. They were able to use new features like the Cloud Run Identity Aware Proxy GA to build a secure, serverless deployment for their applications.

A great partnership with Google Cloud

Colgate and Google Cloud have enjoyed a deep partnership for many years, engaging across many technologies, teams, and design patterns.

They’ve engaged with Product and Engineering across compute, networking, Kubernetes, and serverless as they brought this new way of thinking to their users.

For Colgate and many of our customers looking to address the needs of the modern user, Google’s managed container offerings are a breath of fresh air. Its reliability, scalability, and control offer the  flexibility to build applications that meet the demands of both internal and external consumers. In addition, the variety of container offerings available on Google Cloud — GKE and Cloud Run — allow customers to make app deployment decisions based on the amount of Day 2 operations the users are willing to take on. Platform administrators appreciate the reduced management effort, users enjoy reduced downtime, and developers can simply deploy.

Google Cloud contributors: Rex Orioko, Rachel Tsao
Colgate-Palmolive contributors: Matthew Tattoli, Nicholas Farley, and David Wiser

Are you ready to start your IPv6 journey? Is your cloud provider ready to start it with you? Google Kubernetes Engine (GKE) supports dual-stack Kubernetes clusters to help your journey to IPv6 while ensuring your applications are v6-ready. And to address the operational demands for IPv6 workloads, we’re adding several features to GKE networking to expand protection for both inbound and outbound IPv6 traffic, making them more highly available, secure, and observable. 

The following features for dual-stack GKE clusters are now IPv6-aware, making it easier to enable v6 workloads with solutions that use both v6 and v4 Pods:

Load Balancer Services

FQDN Network Policies 

Dataplane V2 observability

These new features complement the extensive work we’ve been doing for GKE to support IPv6 at the same level as we do IPv4. For example:

Dual-stack clusters – We’ve supported IPv4 and IPv6 front-ends with Ingress for some time, and our managed Gateway API has supported them since it launched. As of December 22nd, 2022, dual-stack GKE clusters have been available with global unicast addresses (GUA) as well as unique local addresses (ULA) on Google Cloud VPC networks. With GKE’s dual-stack clusters, both nodes and Pods get an IPv4 and IPv6 address to enable communication with both IP address families. 

DNS support – GKE supports both IP address families with multiple DNS solutions. From inception, kube-dns supports dual-stack with both A and AAAA records. GKE also provides a more robust, scalable and performant DNS service through Cloud DNS. This Google Cloud-native DNS integration includes in-cluster name resolution with full support of IPv4 and IPv6 records.

Dual-stack Kubernetes Services – For Services, either single-stack IPv4, single-stack IPv6, or dual-stack addresses can be allocated. When we released dual-stack clusters, we supported clusterIP and nodePort Services. These fundamental constructs enable IPv6-capable Kubernetes workloads to be connected in a cluster.

Serving IPv6 to the world – GKE clusters have long been able to expose your workloads in a highly available manner through Kubernetes Ingress services on Google Cloud. By deploying your Gateway and Ingress services on GKE, you get the benefits of Google Networking at the edge to serve and protect with IPv6! Both Kubernetes Gateway API and Ingress on GKE use our tried-and-true Google Cloud Load Balancers, giving you the assurance of proven infrastructure. Additionally, while serving IPv6 to the world, you can protect your applications with Google Cloud Cloud Armor security policies. For example, you can reference your Cloud Armor security policy on your Gateway or the Backend Configuration CRD on your Ingress to define allow and deny lists with IPv6 addresses. 

Now, let’s take a look at the latest IPv6 features and capabilities we’ve developed for GKE.  

GKE Load Balancer Services

We’re excited to announce that the Service type LoadBalancer is now available with dual-stack capabilities. This means you will be able to create Kubernetes LoadBalancer Services and specify their IP families. As a benefit of running GKE, these are deployed as Google Cloud Network Load Balancers, which can be addressed either publicly or privately with the IP address family of your choice (i.e. IPv4-only, IPv6-only, or both). 

Here’s an example of a YAML that you can use to create a dual-stack Kubernetes LoadBalancer Service on GKE exposed as a Google Cloud Network Load Balancer:

Once you’ve created a dual-stack Kubernetes LoadBalancer Service, you can confirm that both an IPv4 and IPv6 address have been assigned to the Service:

You can use the standard Kubernetes API to create dual-stack Load Balancers and apply GKE annotations as you wish. 

GKE FQDN Network Policies

We are advancing our capabilities for GKE with dual-stack support for fully qualified domain name (FQDN) Network Policies. This exciting feature advances the Network Security posture of workloads deployed on GKE to account for IPv6-capable applications.

By leveraging both A and AAAA records, FQDN Network Policies seamlessly provides advanced network security for both IPv4 and IPv6 address families. FQDN Network Policies enforce egress traffic policies when a workload reaches out to specific destinations that are outside of GKE cluster(s) resolving as IPv4 or IPv6 addresses. FQDN is additive to any existing endpoints allowed by the egress Network Policy. Once FQDN Network Policies are created and applied as an egress policy, an implicit DENY is applied for all endpoints that are not specified as an allowlisted destination.

These capabilities provide network security consistency across both IPv4 and IPv6 as you bring your IPv6-capable workloads onto GKE. 

GKE Dataplane V2 observability

Opening up a world of metrics — our GKE Dataplane V2 observability launch brings visibility into your IP4/IPv6 workloads. This feature set includes metrics and troubleshooting tools to make your dual-stack GKE clusters operationally ready. The GKE Dataplane V2 observability stack enables you to have dual-stack Pod traffic metrics for the network info you care about. You can use Cloud Monitoring Metrics Explorer to monitor Dataplane V2 metrics for your IPv6 workloads, while our Managed Hubble solution for IPv6 Kubernetes workloads on GKE lets you troubleshoot the environment. The open source Hubble project is an observability platform built on Cilium and eBPF. Built for GKE’s Dataplane V2, our Managed Hubble UI gives you visibility into connection information and Network Policy enforcement in the form of a service map and a Network Policy verdict table. Finally, a CLI for interactive live troubleshooting lets you better understand your dual-stack Kubernetes workloads. 

Get ready for dual-stack GKE Clusters

We hear from our users that dual-stack clusters are the stepping stones to an IPv6-only world. Together, this suite of features improves the operational readiness of your Kubernetes workloads for IPv6. Going to production with IPv6 implicitly means showing operational readiness in terms of high-availability, security, and observability. These releases should increase your confidence running dual-stack workloads on GKE.

For further reading, check out these resources on our current dual-stack capabilities.

References

GKE dual-stack clusters network overview

GKE dual-stack cluster creation on an IPv4/IPv6 network

Cloud DNS for GKE

GKE Gateway details

GKE Gateway configuration for Cloud Armor security policies

GKE FQDN Network Policies

GKE Dataplane V2 observability

Cloud Monitoring Metric Explorer

Starting today we will be rolling out a new Experimental release of cloud-based maps styling for the Maps JavaScript API to give you more control over the look and feel of your maps than ever before You now have more options to fine-tune your geospatial use cases by targeting more of the content that matters most. More granular control over styleable elements such as labels visibility, areas fill color, and stroke settings, will enable you to better integrate your own design decisions into the map styles you create.

Expanded map features and POIs

Behind these improvements is a new inventory of customizable map features that supports nearly 100 individual map elements, which is two times more features and four times as many POI categories as the current generally available version of cloud-based maps styling. This means you can build maps with our POI data that covers 200+ million businesses and places, and have more control to filter and customize what data you show on your maps. You now get more granular control over the look and feel of your map styles with new cartographic details that were previously not accessible. For example you can now style in a different way areas of reservations, crops, and types of water surfaces. You can also have diverse settings across labels applied to POI categories of tourist attractions, recreational areas, emergency services, retail, and more

Improved stylers

Along with the expanded taxonomy, we’ll roll out some new styling capabilities for different cartographic elements, such as geometries and labels. These new customization properties are all designed to give you even greater flexibility to create custom styles that reflect your brand or application’s unique needs.

How to get started

Cloud-based maps styling is included with Dynamic Maps for Maps Javascript API. Developers can use Cloud-based maps styling features for Dynamic Maps by creating a Javascript Vector Map configured MapID and a new map style in Google Cloud Console.

To try these experimental styling capabilities choose the ‘Experimental’ option as you create a new map style.

Experimental map styles require your web app to use JavaScript Maps API version 3.47 or higher for vector-based maps and 3.49 or higher for raster-based maps. For unlocking most of the experimental features consider JavaScript Maps API beta channel. Over the course of this Experimental phase, we’ll expand the list of customizable map features and related stylers for better usability and greater styling control across all zoom levels.

We believe that the new cloud-based map styling in Google Maps Platform will empower you to create more engaging and informative maps than ever. Try it and let us know what you think!

For more information on Google Maps Platform, visit our website.

In an ever-changing business landscape, the challenges of fraud detection, customer service enhancement, and supply chain optimization present themselves more frequently. With the necessity of real-time analysis growing evermore critical, solutions for these challenges deliver significant value.

For instance, the ability to identify fraudulent transactions as they transpire safeguards the financial health of your business while preserving data integrity. Next, by monitoring customer interactions and identifying potential issues, it’s possible to refine customer service processes, thereby reducing customer attrition. Lastly, vigilance in tracking the flow of goods within supply chains can aid in detecting potential disruptions. This will not only optimize your business’s supply chain operations but also guarantee that products are where they need to be at the right time.

Thus, implementing real-time analysis solutions offer enhanced protection, improved customer satisfaction, and optimized operations, catering to the vital demands of our rapidly evolving world.

In this blog post, we’re excited to showcase a real-time analytics pipeline that effectively utilizes Google Cloud services and Sigma Computing to serve these diverse scenarios.

Architecture

Sigma Computing, renowned for its cloud-native business intelligence and data analytics solutions, empowers business teams by making complex data modeling and sophisticated analysis accessible to everyone, without needing extensive coding skills. By converging Sigma Computing’s data democratization capabilities with Google Cloud’s robust, scalable, and secure infrastructure, we’re serving a wide array of use-case scenarios.

The collaboration of Sigma Computing and Google Cloud brings together the best of both worlds — Google Cloud’s advanced cloud computing infrastructure and Sigma’s simplified data analysis. Together, they enable businesses to make data-driven decisions swiftly, thus promoting efficiency, innovation, and growth.

Pub/Sub: Real-time signals or events are channeled into Pub/Sub, which efficiently processes real-time signals or events. Its value lies in its ability to integrate and manage distributed services and handle large message volumes. It ensures reliable message delivery, high scalability, and facilitates real-time analytics, event-driven computing, and IoT communications. These capabilities make it an asset in instantly gaining business insights, enhancing infrastructure responsiveness, and ensuring secure IoT communications.

Cloud Dataflow: This serverless, fully managed Google Cloud service is tailored for real-time streaming tasks, constructing a metrics processing pipeline from Pub/Sub. It’s invaluable for quick data processing and analytics, underpinning real-time decision-making. Its capabilities include auto-scaling to meet data volume changes and ensuring consistent, reliable performance. Typical use cases encompass live dashboards, anomaly detection, and real-time personalization.

Cloud Memorystore: Cloud Memorystore is used as the Metrics Database for its effective in-memory capabilities, supporting real-time scenarios. It offers built-in functions for diverse needs, delivering high performance and low-latency data access. Ideal for use cases like caching, session management, and gaming leaderboards, it enhances speed, scalability, and application responsiveness.

BigQuery and Cloud Storage: To further incorporate historical data into real-time analysis, you can transition data from Cloud Memorystore to BigQuery using the Cloud Memorystore for Redis API. This allows you to fully exploit BigQuery’s robust analytical capabilities and its scalable and efficient performance. Here’s the general workflow:

Create a Cloud Storage bucket to store the exported data.

Export the data from Cloud Memorystore in Redis RDB format.

Use the Cloud Memorystore for Redis API to copy the data from Cloud Memorystore to the Cloud Storage bucket.

Use the BigQuery Import tool to import the data from the Cloud Storage bucket into BigQuery.

Based on your specific needs, you might also consider streaming data directly from Pub/Sub to BigQuery via Dataflow, while also streaming data into Cloud Memorystore. Should you wish to explore this further, additional information is available in this referenced Google document.

Sigma: Sigma is a cloud-native analytics tool that presents users with real-time insights from large-scale data sets. Its interface resembles Google Sheets, making data analysis and visualization accessible to both code-free and code-friendly users. Despite its simplicity, Sigma can handle datasets as vast as Google BigQuery tables, turning them into interactive documents. It also provides direct, live access to data without extracting or moving it, ensuring strong data governance and security. Overall, Sigma combines the ease of spreadsheets with the power of big data analytics, creating a versatile tool for businesses. For further guidance on integrating Sigma with Google Cloud services, please visit Sigma’s documentation. 

Cloud IAM, Cloud Logging, Cloud Monitoring: These services play a pivotal role in our design. Specifically, 

Cloud IAM ensures secure and granular access control to your Google Cloud resources, enforcing the principle of least privilege and safeguarding your data and services. 

Cloud Logging enables you to store, search, analyze, and alert on log data and events, aiding in troubleshooting and gaining insight into your applications’ operational health. 

Lastly, Cloud Monitoring offers real-time visibility into the performance, uptime, and overall health of your applications, proactively notifying you of any issues so you can respond promptly, ensuring service reliability and optimal user experience.

Conclusion

In conclusion, our demonstrated real-time analytics pipeline offers a dynamic solution to modern-day business challenges, combining the power of Google Cloud services and Sigma Computing. This architecture not only responds rapidly to evolving situations but also integrates historical data, providing a comprehensive view of business operations. With Sigma, even users without coding skills can gain actionable insights from their data. We are confident that this solution will elevate your business decision-making processes, ensuring you stay ahead in this fast-paced world. Try this solution and experience firsthand how it can revolutionize your real-time analytics capabilities. Don’t miss out on this opportunity to upgrade your business intelligence and take your data-driven decision-making to the next level.

Acknowledgements: We sincerely express our gratitude towards David Porter and Reed Rawlings from Sigma Computing for their invaluable suggestions, recommendations, and review, and the Google Cloud and Sigma Computing team members who contributed to this collaboration.