To empower developers to identify sensitive content in a rapidly changing media environment, we are excited to announce Text Moderationpowered by PaLM 2, available through the Cloud Natural Language API. Built in collaboration with Jigsaw and Google Research, Text Moderation helps organizations scan for sensitive or harmful content. Here are some examples of how the Text Moderation service can be used:

Brand Safety: Protect against user-generated content and publisher content that are considered not “brand safe” for the advertiser  

User protection: Scan for potentially offensive or harmful content

Generative AI risk mitigation: Help safeguard against the generation of inappropriate content in outputs from generative models

Promote brand safety

Brand safety is a set of procedures that aim to protect the reputation and trustworthiness of a brand in the digital age. One of the biggest risks to brand safety is the content that ads are associated with; if an ad appears on a website that contains content that does not conform with the sponsoring brand’s values, it can reflect poorly on the brand and organization, so it’s important for companies to identify and remove content that isn’t aligned with brand guidelines or consistent with the brand. 

Text Moderation can be used by our customers to identify content that they determine is offensive or harmful, sensitive in context, or otherwise inappropriate for their brand. Once an organization has identified this content, teams can take steps to remove it from advertising campaigns or prevent it from being associated with the brand in the future, helping ensure that advertising campaigns are effective and that the brand is associated with positive and trustworthy content.

Protect users from harmful content

Digital media platforms, gaming publishers, and online marketplaces all have a vested interest in mitigating the risks of user-generated content. They want to provide a safe and welcoming environment for their users while also maintaining an open and free exchange of ideas. Text Moderation can help them achieve this goal, using artificial neural networks to detect and remove harmful content, such as harassment or abuse. These efforts can help reduce harm, improve customer experience, and increase customer retention.

Mitigate risks of generative models

Over the last year, progress in AI has enabled software to more reliably generate text, images, and video, leading to new products and services that use machine learning, including text generators, to create content. However, with any AI content generation, there is a risk of producing offensive material, even inadvertently. 

To address this risk, we have trained and evaluated the Text Moderation service on real prompts and responses from large generative models. Text Moderation is versatile and covers a broad range of content types, making it a powerful tool for protecting users from harmful content.

Getting started with Text Moderation using the Natural Language API

Text Moderation is powered by Google’s latest PaLM 2 foundation model to identify a wide range of harmful content, including hate speech, bullying, and sexual harassment. Easy to use and integrate with existing systems, the API can be accessed from almost any programming language to return confidence scores across 16 different “safety attributes.”

Visit the Natural Language AIwebsite to give it a try and refer to the “Text Moderation” page for details. You may also try out the Text Moderation codelab here.

Are you expanding your business to a new region using Cloud SQL? Do you need to move your applications that are running on Cloud SQL to a region that has specific data compliance or geographic locality requirements? Maybe you have experienced a regional outage, and you need to move your Cloud SQL instance into another region. Or perhaps you are an enterprise customer looking to migrate to the newly announced Cloud SQL Enterprise Plus edition for the 99.99% availability, near-zero downtime planned maintenance and 3x improved performance. In all these cases, you would first move your data first to a new region and operationalize applications to quickly start serving customers. This article will focus on reducing migration complexity, application downtime, and simplifying the migration process for Cloud SQL. 

This blog post entails steps for planning a regional migration with Cloud SQL:

Assess migration requirements

Migration approach

Application enablement

Assess migration requirements

Whenever a migration plan is considered, you will need to define application-specific integration requirements. These applications interact with backend databases and help define regional migration decisions, for example:

What is the size of the database?
The size of the database determines how long it takes to migrate it. You can control the time to a degree by choosing between an online approach (e.g., replicas) or an offline approach (e.g., export/import or backups) approach. Online approaches are ideal for applications that cannot afford an extended downtime period, whereas offline approaches are simpler to plan and execute as long as you can afford the downtime. 

Is there downtime tolerance?
Applications having Change Request schedules in the production environment make it easy to decide migration complexity. In cases, where downtime is acceptable for change deployment, consider a simple one-time migration approach rather than continuous migration. 

How is the Database connectivity configured in the application?
Cloud SQL regional migration may require updates to the database connection configuration within your applications. If your applications are already using Cloud NAT gateway, Cloud DNS, or a Load balancer etc. for connecting to Cloud SQL, then updating their configuration will be easy without impacting applications itself.

What is the hourly size of updates to your database?
Regional migration approach decisions are highly impacted by frequency of changes or transactions hitting your database every hour. Choosing a migration window with the lowest transaction frequency can help you avoid the risk of losing transactional data, high number of user connection failures, application timeouts, high error rates, etc. This makes the transaction frequency during the hour of migration a critical factor to consider. Identifying the time when your instance handles the lowest amount of traffic to the database can help you plan your migration more easily.

Do you have client side database encryption?
Cloud SQL offers capability to encrypt data that can be decrypted by applications. Encryption keys used for data encryption and decryption are stored in Cloud key management service (KMS). The usage of regional or global KMS keys for data encryption impacts the migration decision for Cloud SQL. In case of a single region KMS key, data might need to be re-encrypted depending on the migration approach used. For more information, refer Cloud SQL interaction with encryption keys. 

Answering above questions helps you plan the best migration path possible and to migrate Client applications to the new instance of Cloud SQL. This ensures that the client applications do not experience a split-brain state.

Region migration approach

For cross-region migration of Cloud SQL databases, there are various approaches available. We will discuss most popular approaches in the following section:

Cross-region read replicas for region migration:

A read-only replica contains an entire copy of the database and can serve reads without requiring a round trip back to the leader region. In doing so, read-only replicas deliver low latency stale reads to nearby clients and help increase a node’s overall read scalability to clients everywhere. Cross-region replication makes it easy to create a fully managed read replica in a different region than that of the primary instance.

The cross-region read replica is set up to asynchronously replicate the primary instance. This means that the read replica can lag behind the primary instance. If the read replica is not in sync with the primary instance during migration, then new data in the primary instance will be lost. To ensure replica is not lagging behind primary instance Monitoring metrics must be defined and tracked on the replica instance during migration hour.

When the whole region needs to be migrated due to region expansion or business requirement, this cross-region read replica can be promoted to become a standalone instance. During replica promotion scenarios, the Cloud SQL instance IP address may stay the same or change. 

With zonal read replicas, the Cloud SQL replica instance IP address will be the same as the primary instance. 

However for regional read replicas, the Cloud SQL replica instance IP address will be different from the primary instance. 

So, when you promote a Cloud SQL regional read replica, its IP address becomes the IP Address of the newly promoted primary instance. Since applications are pointing to IP Address of Cloud SQL before promotion (which was primary instance IP), a config update will be required to point to the IP address of newly promoted Cloud SQL instance (which was regional read replica IP). 

The complexity of repointing applications to the new Cloud SQL instances depends on how your application configuration is defined. Refer Cloud SQL lab to understand how to use Cloud DNS for config update.

Data migration service for new regional Cloud SQL instance:

Database Migration Service (DMS) can be used to migrate databases from an existing CloudSQL instance to a new one with the same or different Cloud SQL edition in a different region, and then perform an ongoing replication from the source database into the target. DMS supports CDC-style migrations for all sources (both homogenous and heterogenous migrations) which continually move changes from the source to the destination. See the PostgreSQL DMS known limitations.

Backup restore for new regional Cloud SQL instance:

Automated or on-demand backups stored on multi-regional buckets can be used to migrate Cloud SQL to another region. Setting up Cloud SQL instances from backup consumes some time as it involves creating a new instance and then restoring from the backup. 

Export and import database in new region:

Database exports can also be used for database migration to a different region. Critical considerations while choosing export/import databases are mostly around efforts, time, and data loss tolerance levels for an organization. For regional migration, one time export and import can be used in multi-regional Cloud storage buckets which can be accessed in the target region. 

To plan disaster recovery using export/import, the process should be automated to export databases regularly based on business requirements. For example, what are your SLA requirements? Are there any compliance requirements such as GDPR to be considered while exporting data? What should be the frequency of data export?

Below is a quick comparison between backup/restore and export/import:

Application enablement

Applications consuming data from Cloud SQL have a dedicated configuration for the database. The complexity of moving away from one configuration to another depends how the configuration is defined. Database connection endpoints are defined in the application runtime configuration. 

It is recommended to use aliases for database endpoints instead of using database IP addresses in the connection strings. This allows for database migration without impacting applications, as there is no need to change the connection string. You will only need to update the alias details of the migrated Cloud SQL database. 

Enterprise customers mostly use a private Cloud SQL instance unless there is any specific need for a public instance. For example, Cloud DNS can be used to resolve DNS records (FQDN) to the public or private IP address of Cloud SQL. The corresponding Cloud DNS name can be used in application configuration for pointing to the SQL host record. After migration, it will only require an update to the Cloud DNS record for the migrated Cloud SQL instance.

Learn more

Cloud SQL migration from one region to another region involves the analysis of database components impacted, application enablement and reconfiguration, network configuration, impact on user interfacing applications, migration of database monitoring, and many more activities. We discussed different approaches for Cloud SQL regional database migration using backups, read replicas, and offline exports. Using cross-regional read replicas is the most convenient approach among all, however the decision should be made based on business, cost and latency requirements. We hope that this blog has helped you better understand the differences and choose the configuration that is right for you.

To get started, create a Cloud SQL instance in your GCP account or in a SQL regional migration qwiklab to try the regional migration experience.

Want to learn more? Check out the resources below to dive in:

Cross-region replica for Cloud SQL

Scheduling Cloud SQL exports 

Database Migration Program

You do not have to burden yourself anymore with a long list of manual steps required to setup an application locally, test with an in-memory database, sift through discrepancies between environments and SQL dialects or not being able to test network failures typical for a distributed system.

When building Twelve-factor Google Cloud applications, developers are looking for a consistent way to develop, debug and test locally with the same containerized application dependencies as they would run in production in GCP, the environment parity from Dev to Prod – factor Ten on the list of twelve-factor apps.

Google’s Java Cloud Client Libraries and Spring Cloud GCP make migrating, or building a new cloud-native, production-ready Spring Boot application a breeze, with developers able to quickly spin up the application and focus on building out the business logic.

In this blog post we’ll focus on significantly improving (and speeding up) your development experience for Google Cloud Spring Boot Java Applications, with a consistent way to develop, debug and test locally, following the goal of environment parity from Development-to-Production in Google Cloud.

Google Cloud Emulators, Testcontainers and Spring Boot to the rescue

Testcontainers is an open source framework for providing throwaway, lightweight instances of databases, message brokers, web browsers, or any other custom Docker container.  There is no more need for mocks or complicated environment configs. Testcontainers allow you to define your test dependencies as code, then simply run your tests and containers will be created and then deleted at the end of the test lifecycle.

Google’s Cloud SDK provides libraries and tools for interacting with the products and services provided by the Google Cloud Platform. Testcontainers modules are preconfigured containerized implementations of various dependencies that make writing your tests very easy.

Currently, Testcontainers Google Cloud Module supports Bigtable, Datastore, Firestore, Spanner, and Pub/Sub GCP Emulators in Java and other languages.

Spring Boot 3.1 has added Testcontainers support, allowing you to configure all application dependencies (databases, messaging systems, caching mechanisms, dependent services) to start automatically when you run or test the application locally.

If you are looking to learn more about how to test your Spring Boot applications with Testcontainers, follow this Getting started guide.

Development tasks

Before we dive into details, let’s revisit the some of the tasks you would usually address as a developer:

Write Unit Tests for individual pieces of functionality, say classes/methods

Write (Domain) Service Integration Tests to test domain operations and business rules, including the Integration with dependent applications or cloud services

Write Application Tests to functionally test your application and its web controllers 

Write Network Failure Tests to assess that the source code will handle any such failure, part of chaos engineering best practices

Local Development with app dependencies

Blog Source Code

This blog post is supported by a full set of Spring Boot sample apps, with tests highlighting the usage of Cloud Firestore and open-source Postgres (for Cloud SQL) Testcontainers.

Clone the Git repository and follow along in the Audit (Firestore) and Quotes (Postgres) services to gain hands-on experience with the concepts exposed herein. The following material will continue on the Firestore usage path.

Test configuration 

First, add the spring-boot-testcontainers as a test dependency to your Maven pom.xml:

Next, add the Google Client Libraries Bill-of-Materials(BOM) for compatibility across more than 200 Google Java libraries:

Last, add the dependency on the respective application dependent service:

To test with the testcontainer for the Firestore emulator, configure the @Container and provide its dynamic properties in the test class. 

Ex: Firestore Service Test [test]

Note that Spring Boot 3.1  has added the concept of Service Connections, which automatically configures the necessary Spring Boot properties for the supporting containers. 

A Postgres configuration would be reduced to:

Writing (Domain) Service/Integration Tests

Testing your service against a containerized Firestore emulator provides the confidence that the code will run smoothly in Cloud Firestore: [test]

Writing Application Tests

Writing application tests allows you to test your app as you would run it in a cloud environment in Prod. 

For example, your service is running in a Serverless environment and is a component of an event-driven architecture. CloudEvents is a specification for describing event data in a common way and you can now test your HTTP call with CloudEvents in the web controller, while leveraging Firestore testcontainers: [test]

Writing Network Failure tests

Network failure conditions between services running in the cloud present one of the highest risks in distributed software, since they directly affect service delivery. Before running your services in cloud environments, you want to test the resiliency of your service client code in your codebase for any such random disruptions, which can cause applications to respond unpredictably and break under pressure.

These tests are part of Chaos Engineering best practices.

To simulate network failure conditions, you can place Shopify’s Toxiproxy container in between test code and a container, or in between containers. [test]

You can now simulate timeouts due to latency, retries or dropped network connections. 

For example, you can test with a simulated timeout as follows:

Local Development with application dependencies

Starting with Spring Boot 3.1, you can also use Testcontainers during your regular dev time for running and debugging the application locally with your respective dependent services! 

There is no need to do any extra work, just configure a TestAuditApplication and TestcontainersConfig class in your test classpath under /src/test/java and start your app.

Note that the app is started with the TestcontainersConfig class attached to the application launcher using .with(TestcontainersConfig):

 You can run the TestAuditApplication from your IDE or simply with:

Summary

By using Google Cloud Emulators, Testcontainers and Spring Boot 3.1, you can significantly improve (and speed-up) your development experience for Google Cloud Applications, with a consistent way to develop, debug and test locally with the same containerized application dependencies as you would run in production in Google Cloud — the environment parity from Dev to Prod.

Check out the codebase, if you want to get more hands-on experience with Spring Boot Applications using Google Cloud Emulators and Testcontainers. 

For questions or feedback, feel free to contact me on Twitter @ddobrin.

If it isn’t already clear, Google Cloud is invested in being FinOps-first. At Google Cloud Next 2023, we made some pretty exciting FinOps product announcements for Google Cloud Billing. Here are the top five:: 

1. Google Cloud continues to pioneer the FinOps movement: Through our strong roots in the FinOps Foundation, we announced our contributions to v 0.5 of the FinOps open billing standards FOCUS (FinOps Open Cost & Usage Specification) project. V 0.5 provides a common language and set of definitions for data dimensions, metrics, and attributes regarding billing data. Datasets conforming to FOCUS will enable FinOps practitioners to perform common FinOps capabilities, using a generic set of instructions, regardless of the dataset origin. 

Why do we want to participate in a project designed to level the cloud billing data playing field across all cloud service providers (CSPs)? First, being open is a part of Google’s DNA, having contributed to foundational open source internet technologies over the past 20 years. Second, we believe that customers who innovate faster actually save money faster. We don’t want you spending time trying to decipher your cloud bills! We want you to leverage your cloud billing data to make better business decisions that fuel innovation, drive industry changes, and leapfrog your competition. Open, high-quality cloud cost data is non-negotiable. 

In support of good quality cloud cost data, Google Cloud Billing announced support for App Engine, Spanner, Firestore, and BigQuery granular, resource-level cost data that comes out of the box with theBilling detailed big query data export. 

To further FinOps industry adoption, we also launched a spotlight on FinOps thought leadership series on the Transform with Google Cloud thought leadership blog. In this monthly series, learn how FinOps has been adopted through real-life illustrative examples at work today from industry leaders.  

2. DuetAI is trained to a sample Certified FinOps Level. As part of Duet AI in Google Cloud, FinOps practitioners can also take advantage of AI integrated assistance for Billing based on models trained on Google Cloud best practices. Tailored for FinOps practitioners, you can use Duet AI to increase your Google Cloud billing proficiency. To measure Duet AI’s FinOps IQ, we partnered with the FinOps Foundation to put Duet AI to the FinOps test — literally: Duet AI passed a similar exam to the FinOps Certified Practitioner with flying colors. You can try it for yourself here. We have a robust roadmap for Google Cloud Billing has to unlock the power of generative AI that prioritizes accuracy first, but this is just the start. Look out for more innovations in this area. 

3. New FinOps Hub helps you mature your FinOps practice with smarter cost optimizations, new FinOps score, and peer benchmark: We’re focused on removing the cognitive load for cloud Finops practitioners . But when it comes to optimizing cloud spend, one of the hardest parts is knowing where to start. That is where the FinOps hub comes in.

We announced theFinOps Hub public previewat Next ‘23 to provide one pane of glass to identify, adopt, and measure all cost optimizations across an entire Google Cloud landscape. WIth advanced machine learning capabilities, the FinOps hub tells users what their FinOps score is — a tangible measure of how they are using Google Cloud billing data and tools to achieve maximum cloud value, with suggestions on how to improve. Peer benchmarks (with an opt-out option) provide context with respect to how users are performing with their FinOps practice and overall rate optimizations through committed use discounts (CUDs). 

Additionally Google Cloud cost optimizations have also gotten a lot smarter — they’re no longer just about rate optimizations like purchasing a CUD or rightsizing a VM; rather, we recognize that cost optimizations should be orchestrated across your entire cloud landscape, doing both rate and usage cost optimizations together through personalized prescriptions, always with the ability to personalize optimizations according to business needs. The FinOps Hub includes rate and usage optimizations for a variety of Google Cloud services. Finally, the FinOps Hub provides a realized savings ledger so cloud FinOps teams can track their return on investment (ROI) on all their cost optimizations in one place, easily.  

4. Budgets gets extended to engineers: We also announced a great new tool for project users to keep track of their cloud costs through the extension of the rich Cloud Billing budgets tooling. Project owners, Project Editors and Project Viewers can now create budgets and Project owners get notifications about actual or forecasted budget alerts in the Google Cloud console or through the Cloud Billing Budget API.

5. Google Cloud Billing is building for FinOps 2.0: We’re building towards a future of FinOps automation, as seen through our latest API enhancements to get a head start including the new Pricing API in public preview. The new Pricing API allows you to not only get list prices on all the available services but also account-specific custom prices and discounts, with visibility into any internal SKUs that are available to your billing account as part of your contract or agreement. The Pricing API also provides additional metadata such as SKU to service mapping, a list of SKU groups, SKUs to SKU group mapping, and a geo and product taxonomy that allows you to further slice and dice the SKUs for improved analysis. This API, along with others such as the Commitments API, provide the basis for customers to advance their own advanced cost management applications. Learn how SAP leveraged Cloud Billing APIs and data in their journey to controlling costs and building financial resilience with automation through their Google NEXT session here. 

To stay up-to-date on all things cloud FinOps and Google Cloud Billing, sign up for our Billing Advisory cohort at billing-cohort@googlegroups.com, where you can gain access to early features, participate in user feedback sessions, and learn about our roadmap with a group of like-minded cloud FinOps professionals.

Editor’s note: This is the third of a three part series from the BCW Group, a Web3 venture studio and enterprise consulting firm serving enterprise clients who want to integrate existing products or otherwise branch into the Web3 space. Today’s blog discusses the frameworks used for creating one-click, end-to-end blockchain deployments into Google Cloud. You can read the first post here and the second post here.

Google Cloud’s BigQuery public datasets offer a window into the world of data-driven insights and analysis. These publicly available datasets offer a diverse array of domains, including economics, genomics, blockchain, and more. Researchers, analysts, and curious minds can penetrate datasets with the massive processing power of BigQuery. Whether unraveling complex trends in global trade, unlocking the mysteries of the human genome, or visualizing geographical patterns, the BigQuery public datasets empower users to find the story in the data.

BCW Technologies is working to create a framework that will help organizations and individuals access existing extract-transform-load (ETL) codebases and Google Cloud public datasets platform (PDP) so they can build a one-click, end-to-end, deployment into Google Cloud. We will be working primarily on the blockchain side of the PDP. These datasets can support many data strategies including academic inquiry, supplying historical chain data to users, or building out a comprehensive Big Data as a Service (BDaaS) platform.

Blockchains such as Ethereum or Bitcoin already generate large amounts of data. However, when discussing high-throughput chains such as Solana, building an ETL is not enough: developers must be cognizant of the entire pipeline. Since all states assist in validating all subsequent states, a comprehensive set with no trimmings is needed. Developers must consider all aspects of the extract-transform-load (ETL) process while calibrating their own deployments to keep up with these networks while designing a system that can be deployed and integrated by a wide array of participants.

The Blockchain ETL platform is designed to be a modular framework. It extracts blockchain data from a data source (such as an RPC node), then packages it using the efficient Protocol Buffers format before sending it to a data loader. This data loader first transforms the data to match the data storage schema before performing the insertion. In this case, the storage is Google Cloud BigQuery, which is well suited for the many terabytes of blockchain data. Given how much data is moving through modern blockchains, BQ’s large insertion support as well as scalable queries make it more than ideal for warehousing this data. Consequently, this makes the public dataset useful for use cases such as data science, comparison with other blockchains, or indexing historical on-chain actions.

The Solana blockchain has one of the shortest block times, thereby creating more frequent new blocks and blockchain state changes compared to other chains. This presents unique challenges. In this case, and pertaining to any blockchain network built on a rapid consensus mechanism, the data that is generated can quickly become unmanageable on the decentralized ledger where it is recorded.

BCWT seeks to build out an end-to-end, one-click, solution to the rapidly mounting level of data that highly performant chains generate. Authorizing configurations & submodules into an infrastructure-as-code deployment is important for the mission of assisting in democratizing the PDP data set (public is in the name afterall). By defining configurations that are compatible with both BCWT and community authored ETL, we seek to build out a solution that leverages the existing codebases, alongside our performance-oriented architecture.

The Solana ETL system consists of three primary parts with accompanying modules: a non-voting validator for keeping the state up-to-date, the transformer, and a Google Cloud BigQuery instance. The validator connects to the Solana network and follows up to three days worth of full state data (account snapshots included). Next, the transformer acts as a middleware between the data source and storage, and is deployed as a separate Compute Engine instance. Finally, Google Cloud BigQuery accepts the structured data.

BigQuery offers a tremendous amount of power for both querying and inserting data, but there is also the opportunity to be thoughtful in design to help control cost. By defining efficient partitioning routines and queries, selecting an appropriate egress point, and defining tables that map to your sources, controlling costs is possible. We are currently testing and authoring a collection of Solana-flavored repositories which can be seen here.

A highly performant programming language that can handle the immense data churn and scale must be used to compliment the aforementioned requirement of large-scale data in creating the ETL. In our case, we chose the highly performant systems language, Rust. This is also the implementation language of Solana, and offers speed, memory, and stability benefits resulting from its unique memory management, compilation optimizations, andzero-cost abstractions. Additionally, Rust’s support for conditional compilation has enabled us to design our codebase with modular blockchain support, such that we can add future blockchains in a plugin design. With each of these future blockchains, we plan to support the open source community with publicly available Rust packages to avail other developers direct library access.

One of our goals with Rust is to keep our code within community expectations to encourage open source collaboration. For example, we use well-known Rust libraries, including clap for our command-line interface, reqwest for making HTTP requests, Serde for deserializing JSON RPC responses, and Tokio to parallelize our extraction process. We also attempt to use functional programming as much as possible to accommodate data engineers interested in the project.

Upon deployment and initialization, the Solana ETL will first begin extracting data from Solana Labs’ instance of Google Bigtable (BT). It will also spin up a Solana Node and then begin working up from the genesis block, filling a designated BQ instance. Three data requests are necessary to extract data for the pipeline: (1) blocks, (2) accounts, and (3) token mints. The block data response includes all of the necessary data to populate the transactions, instructions, blocks, block rewards, and token transfers tables. However, Solana Labs’ BT instance only provides the block data, so an RPC node is required for the account and token data. Also important to note is the order of the data requests: account and token data is only requested when a transaction in the block creates an account or token mint, so the block data must be parsed first before these final requests can be made.

Initially, there will be a testing period, where Google Cloud will host a test dataset for selected Solana ecosystem participants and companies to engage with. Next, the overall apparatus will be deployed into the PDP organization and the dataset will be made accessible to all ecosystem participants. Finally, all code and deployment repositories will be added to the Blockchain-ETLrepositories for access by developers. These repositories will also include accompanying helmcharts, IAC configurations, and playbooks for rapid deployment into a GCP account.

While cloud technologies are often considered a centralizing force in decentralized technologies, it would be unrealistic for many to access the full scope of data that is generated by Solana. Archive nodes that handle both the full scope of all Solana blocks, accounts, and transactions are hard to find and access while including Geyser-enabled state snapshots is a very heavy lift for any one team or individual. By opening up access to a public data set in a highly performant infrastructure, Google Cloud helps to solve these challenges and make Web3 more accessible to traditional Web2 developers, companies and businesses that are all already using GCP every day. Google Cloud’s commitment to opening data access fosters innovation and discovery, inviting users worldwide to embark on exciting journeys of exploration within the boundless realm of data science.

Today we hear from Freestar, whose ad monetization platform helps digital media publishers and app developers maximize ad revenue with scalable, efficient targeting opportunities. Founded in 2015 and reaching over 70% of U.S. internet users monthly, the company is now among the top 15 in U.S. internet reach. Vijaymohanan Pudukkudi, Vice President of Engineering – Architecture at Freestar, guides the company’s Ad Tech platform architecture. In this blog post, Vijay discusses Freestar’s decision to adapt to a multi-regional cluster configuration for its Memorystore deployment and leverage Envoy proxy to facilitate replication.

As Freestar has continued to grow, we support a global customer base that demands the lowest possible latencies, both from a customer experience and revenue standpoint. Specifically, we aim for ad-configuration delivery with a client-side latency under 80 ms at the 90th percentile, with server-side round trip time for requests under 20 ms at the 95th percentile.

Since Freestar expanded globally, we’ve faced challenges with latency. Originally, we served ad configurations from a single region in Google Cloud’s US-Central region. This resulted in high latency for customers accessing the configuration from outside the US. One solution was to use Google Cloud’s Content Delivery Network (CDN), but it was deemed unsuitable because of the dynamic nature of configurations. To meet our ambitious goals, we had to evolve our single-region architecture (see IMG A.1) to be multi-regional.

The problem

As our global footprint grew, our risk analysis pinpointed a significant technical concern related to our critical service delivery infrastructure. Prior to transitioning to a multi-regional design, Freestar’s infrastructure was concentrated in the US-central region, spread across multiple zones. It left us exposed to potential regional disruptions in Google Kubernetes Engine, which could have led to a complete service downtime, affecting all our customers.

We also rely on Google Cloud’s Memorystore for rapid configuration lookups. When we transitioned to a multi-regional setup, additional network overhead was introduced when reading configurations from the Memorystore cluster located in the US-central region from other regions. When accessing the service from regions outside the US-Central, we observed an additional latency of 100 to 240 ms at the 95th percentile, and the total service latency would have increased from our benchmark of 80 ms to approximately 300 ms. Although this performance might be satisfactory in many scenarios, it didn’t meet the requirements for our specific use case. You can view the related data on inter-regional/continental network latency on Google Cloud in the publicly available Looker Studio Report. To address the issue, we decided to distribute the Memorystore cluster regionally.

The challenges

Currently, Memorystore does not support out-of-the-box replication across multiple regions within Google Cloud. One solution we considered was building application-level code to actively call individual regional API endpoints and trigger refreshes of Memorystore keys. However, this approach would have introduced multiple points of failure and lead to inconsistent configurations across regions. This inconsistency would then necessitate a process to ensure customer Ad configurations were kept consistent across all regions and refreshed as needed. Furthermore, this option would require a significant amount of development effort. Thus, we sought a different approach.

The solution

Ultimately, based on this solution blog, we chose to go with Envoy proxy to facilitate replication. This strategy was essential in enabling high-throughput applications and multi-regional access patterns on the Memorystore infrastructure. Motivated by this approach, we devised a solution to replicate and keep configurations consistent across multi-regional Memorystore clusters.

First, we configured the Envoy proxy to propagate all the write operations to every regional cluster. This procedure is triggered whenever an event occurs in the primary Memorystore cluster in the US-central region. The Envoy proxy is configured as a sidecar container, running alongside the microservice that initiates the Memorystore write operations.

This approach required minimal developer time because the only change necessary in the application code was updating the Memorystore configuration to utilize the Envoy proxy port and IP address. In this case, we used localhost, as the Envoy proxy, and the application ran within the same GKE Pod. Going forward, this solution will also enhance our ability to scale multi-regional clusters without requiring any application code updates.

Envoy proxy sidecar configuration

Remote cluster configuration

This cluster configuration (refer IMG A.2) defines remote regional memorystore IP address and port details. Envoy uses it to interact with the actual Memorystore cluster from the virtual configuration.

Local proxy cluster configuration

Following configuration (IMG A.3) is the us-central virtual / proxy configuration within the pod where the sidecar is running so that it can propagate or mirror the writes commands.

Envoy proxy listener configuration

The Envoy Listener (IMG A.4) initiates a process based on triggers from the application, in this instance, the Config Dispensary. The application interfaces with Memorystore using a local host and port 6379. Following this interaction, the system employs the ‘us_central1’ remote configuration, directing the command towards the actual cluster.

Examining the configuration closely, you’ll notice a specific block configuration set to mirror all commands into another local proxy cluster. Additionally, there’s an exclusion configuration designed to prevent all read operations from being mirrored, thereby restricting such actions.

The image below (IMG A.5) shows the actual configuration settings. It demonstrates how the application from the us-central region (primary location) interacts with Memorystore through Envoy proxy for write operations.

The architecture diagram (IMG A.6) below shows how the regional Memorystore cluster and the microservices communicate. Except in ‘us-central’, microservices function without a proxy. In ‘us-central’, the proxy oversees read/write activities within the region and distributes writes throughout other regions.

The fully configured envoy.yaml and Dockerfile configuration for the Envoy Proxy are available on GitHub.

The architecture diagram (IMG A.7) below provides a comprehensive view of our microservice infrastructure. We leverage services such as Cloud Spanner, Cloud SQL, and Datastore to store the primary copy of our ad configuration in the US-central region. Any changes to this primary configuration instigate a Memorystore refresh in the US-central region. Subsequently, these updates propagate to other regional Memorystore clusters with the assistance of the Envoy proxy.

What comes after ‘Go’: Post-deployment analysis

By switching to a multi-regional Memorystore Redis setup with Envoy proxy replication from a single-region Redis cluster, we’ve significantly reduced latency, matching the performance of the previous single-region deployment.

Though write replication takes around 120ms at the 95th percentile to synchronize data across regions, it’s inconsequential due to our system’s read-heavy characteristic, making the latency of write operations negligible.

Since implementing this setup two months ago, we’ve had no replication or Envoy proxy issues. The application uses the US-Central region as its main region and uses Envoy for all Memorystore operations. Memorystore handles about 40K requests per second at peak hours with 5ms latency at 95 percentile. At peak periods, our Ad Config Service efficiently handles more than 15K API requests per second. Achieving a consistent 80ms latency on the client side (web browser) at the 90th percentile, we’ve successfully reached our targets following the expansion of our critical services across multiple regions.

Collaborating with the Google Cloud team, we’ve turned complex challenges into a robust solution, and the Google Cloud team’s meticulous validation of our system made us confident that it complies with industry standards. We now have a reliable solution for replicating Memorystore data globally, and consider the collaboration with Google Cloud a great success.

With the global energy crisis driving a dramatic spike in the price of fossil fuels, countries around the world have urgently stepped up their transition to renewable power to reduce their reliance on imported, carbon-intense fuel sources. According to the International Energy Agency (IEA), renewables will account for over 90% of global electricity expansion over the next five years, overtaking coal to become the largest source of global electricity by early 2025. 

However, while investment in renewables improves countries’ long-term energy security, the rapid transition presents energy-security challenges of its own. As more renewable energy sources and Distributed Energy Resource (DER) assets (such as rooftop solar panels and microturbines) are added to the grid, this can lead to network congestion and imbalanced energy distribution, affecting the quality of supply. 

A new business model for a changing industry

An industry leader in energy management solutions for more than 125 years, Landis+Gyr helps utility companies and grid operators to manage their energy infrastructure more efficiently. For Landis+Gyr, the answer to the challenges presented by a rapid transition to renewables lies in modernizing the traditional smart metering business model by using artificial intelligence (AI) and machine learning (ML) to make the most of the huge amount of data that its utility customers are collecting every day. 

However, Landis+Gyr initially found it difficult to innovate at the pace required by the rapidly changing energy industry, as its legacy on-premises analytics infrastructure lacked AI and ML capabilities. 

“We wanted to build a new product that enabled advanced analytics, including AI and ML capabilities, which we could offer as a service to our customers,” says Antonio Hidalgo, VP Digital Solutions at Landis+Gyr. “We also needed a solution that was easy to set up, scalable, and which required very little management from our side. Those were the key criteria for us and Google Cloud met them all.”

Google Cloud partnered with Landis+Gyr, making it our mission to help the company to innovate faster with a unified and intelligent data platform

Keeping the lights on with real-time data insights

Thanks to Landis+Gyr’s smart-metering infrastructure, utility companies all over the globe retrieve granular interval data for millions of devices every single day, creating a huge volume of data that needs to be uploaded and processed. Now, with the help of Cloud Storage, BigQuery, and Looker, Landis+Gyr can provide its utility customers with a comprehensive data lake and a complete analytics solution by using a cloud data warehouse, enabling them to make full use of their data to gain powerful insights in real time.

“BigQuery was super fast to set up and it scales seamlessly up and down, which makes the pricing structure very scale-friendly,” says Hidalgo. “Flexibility was also key for us. Our utility customers want to be able to define their own dashboards on a continuous basis, based on the information that is relevant to them. With Looker, we are now able to build and release something in a matter of days, not months. With our on-premises solutions, that was impossible. But on the cloud, with a modern CI/CD pipeline and the right level of automation, we can do it.”

Landis+Gyr is now able to provide its customers with data insights to manage all aspects of the power supply and distribution more effectively, from determining the quality of energy supply, through forecasting energy demand, to helping utilities maintain a distribution network with a growing volume of DER assets.

“Near real-time monitoring of sensor data streams enables our utility customers to predict many network issues in advance,” says Hidalgo. “By using Pub/Sub, we can ingest events and stream them in real time to BigQuery with Dataflow. Utilities can then analyze the network data to identify the problem and make decisions accordingly.”

Mapping the power grid with machine learning

Landis+Gyr is also making use of Google Cloud ML tools to improve the way utility companies maintain an accurate network connectivity model. Understanding the power system’s network topology, or how the various components such as generators, transformers and transmission lines are connected, is crucial for maintaining reliable and efficient grid operations, such as network planning, outage management and fault detection. This topology is traditionally maintained using Geographic Information Systems (GIS), which Hidalgo says are prone to inaccuracy. 

“On average, GIS systems typically have 5-20% inaccuracy in terms of where the end consumers are located in relation to the network,” Hidalgo notes. “With the introduction of new ML capabilities with Vertex AI, we can develop something far more accurate without the need to go to the field to confirm anything. A prediction with 97-98% accuracy is far more valuable from an operational point of view than the current GIS systems.”

Secure data for energy security

As data becomes increasingly important to utility companies, Landis+Gyr understands how critical it is to secure that data, and has been working with Google Cloud to apply cloud security best practices as a cornerstone of its new analytics solution. 

“Before partnering with Google Cloud, we made a careful assessment of the data protection safeguards and security compliance of the different cloud providers,” says Hidalgo. “Google Cloud offers us state-of-the-art data protection and security standards, which means that our customers are the only ones who are accessing their data.”

Future-proofing the industry for a greener tomorrow

The past two years of this partnership have built the foundations for future innovations, not only from a technology perspective, but also from an organizational and operational perspective. And as the transition to renewables continues to transform the energy industry as a whole, Hidalgo is convinced that Landis+Gyr can use ML and AI to transform the way the industry meets the ongoing challenges of this transition. 

“With Google Cloud, we are engineering a scalable, flexible solution that enables our customers to release new features quickly, to meet the fast-changing conditions we are now facing in the energy sector,” Hidalgo says. “And with AI and ML, we have the opportunity to create completely new insights with much more accuracy in terms of predictions, providing an entirely new way for the utility companies to operate the grid. ”

We all know how important the transition to renewables is. That’s why we are proud to partner with Landis+Gyr to lead the data transformation in the utilities sector and ensure that utilities can manage the security of supply with a powerful analytics solution.

Engineering organizations need to be able to quickly and easily identify and resolve technical issues, but this need often takes a backseat compared to other priorities. With so many things to track, it can be challenging to know which ones need alerts. It is possible to set up too many alerts, create too much noise, and lose relevance when faced with real problems. Because of this, we need to clarify why we are creating an alert in two respects:

the relevance of what we’re monitoring to our business goals

the intended outcome once we notify someone there’s an issue

Understanding the relevance of what we’re monitoring can help us support triage in advance. When an issue occurs, we should have enough context to gauge its impact on users, potential cost, and what other problems it can take priority over. Relevance also helps us identify which signals warrant setting objectives. For example, latency is incredibly relevant in a customer-facing application that serves real-time data — yet irrelevant in a weekly cron job to clean up old files across systems. Relevance based on context helps us answer why, what, and when to alert.

The outcomes we intend from an alert notification vary widely, from immediate response to situational awareness. We should not be waking up our on-call engineers for minor issues that automation can already fix. We should also not be tucking away all of our alerts in emails since the more critical issues could slip by. We are determining the outcomes we want and the severity of impact, which can help us figure out who to alert and how.

Why should I create alerts for only some things?

When I create an alert for something, the assumption is that someone can and should act if notified.

Think of alerts you might experience in everyday life:

A flash flood warning on our phone

A text message about a declined credit card transaction

An email about a “login from a new device”

Each of these alerts could exist on a website or in a log, but we’re getting notified directly about each one. We often also get instructions about what to do:

Avoid roads, stay at home, and prepare an emergency kit

Agree that you intended to make the transaction and try again or report it as fraud

Trust the new device, or block access and immediately change your password

In some cases, even if we feel the alert isn’t relevant or are already aware of an issue, we still see the value of having the alert on in general. And even if we need to do some deeper digging, e.g., looking up the store’s name where the transaction took place or verifying the device matches what we’re using, the alert is a good starting point.

The zombie strategy of alerting, where too many metrics have alerts, potentially leads to a situation where the critical issues do not stand out. The saying goes, “If everything is important, nothing is important.” 

Imagine getting a flash flood warning for every flash flood on Earth, not just the ones in your area. Imagine you get a text message not only for transactions, but for every time your bank offers a new type of credit card that might be interesting to you. How about an email every time you log in, even from the same device? These would condition you to ignore the more relevant warnings or otherwise distract you with noise.

We have a starting point: we should have alerts for things insofar as we can fix, influence, or control them, and insofar as not knowing about the alert could be worse for us. Still, that leaves a large set of possibilities that we need to trim down, starting with what to alert on.

What should I alert on?

No matter how much we automate, the constant area of focus is on relevance to users. InWhy Focus on Symptoms, Not Causes, we explored why user-facing symptoms are a better area of focus for alerting than causes.

We should alert on things that are actionable and relevant to users. The SRE Handbook provides examples of service-level indicators that matter to users based on what they expect from your system. A few SLIs of note:

Availability. 500s, unintentional 400s, hanging requests, redirects to malicious sites–all count. Whether it’s the entire site or a small third-party component, anything that disrupts the critical user journey should be considered “unavailability.”

Latency. Fast (as long as it’s humans that are waiting).

Integrity/durability. The data should always be safe. Even if the data is temporarily unavailable, it should be correct when it comes back.

To set a discreet objective for each SLI, hence an “SLO,” you are assuming a threshold where the user starts to notice a degradation in service, or starts to feel that they are not getting what they were promised. Tostandardize the process of setting SLOs, you need to model user journeys and grade the importance of each interaction. 

Starting from “symptoms” your users can feel, you may occasionally discover that there are still “causes” in your system that you need to alert on–a leading indicator deep within your stack that, while its behavior may be invisible to your users, can still impact them down the road. 

Ideally, you’ll move to a state where you can automate responses to any of these issues. A great way to frame your goals for alerting is to start with the question: “What would it take for us to only alert on SLOs?” The gap analysis that follows will reveal where a system is too brittle, or where an organizational process falls short by requiring too many manual operations to respond.

What if I don’t have time to set SLOs?

Setting SLOs or doing a deep analysis of critical metrics takes time and organizational maturity, especially if a service is already complex. To help find a starting place for alerts and dashboards, Cloud Monitoring has anIntegrations Portal with over 50 observability bundles. The bundles include the top metrics, sample alert policies, and sample dashboards to get started with popular Google Cloud and third-party services. 

From this starting point, a team can configure the alert trigger to suit their needs, and use this starting point to extend alerting down the road.

An essential rule of thumb to keep in mind is that alerts are only the “start” of an action or an investigation; they do not form the entirety of your strategy. Not all metrics should have a corresponding alert. If a system grows and becomes more automated, it would be normal to monitor more and alert on less.

Metric-based and Log-based Alerts in Google Cloud

Now that we have a sense of the criteria for picking top-level indicators, how do we actually collect them from our system? What form do they take in Google Cloud?

Metric-based alerts are triggered when a metric crosses a threshold. For example, you could create a metric-based alert that triggers when the CPU usage of a particular instance exceeds 80%. Metric-based alerts also can apply to metrics derived from logs, such as the number of times an error message appeared.

Log-based alerts, on the other hand, are triggered when a specific message appears even once in a log. For example, you could create a log-based alert that triggers when the message “Error: Database connection failed” appears in the logs of a particular service.

After finding the most relevant indicators in metrics and logs, it is necessary to determine the thresholds to trigger alerts.

At what point should I trigger an alert?

Not all issues have the same impact, and often, the impact changes over time or intensity. For this reason, teams need to be able to pick and adjust thresholds for when to trigger an alert, and may choose to create multiple policies for the same signal based on varying degrees of severity. 

The duration and breadth of an issue affects the impact. For this reason, alert policies should be configured to match “how widespread” or “how long” a team determines an issue would be impactful. (Incidentally, this is exactly the type of estimation that goes into designing and tuning SLOs.)

For example, if you’re alerting on CPU utilization on your instances, do you want to get alerted for every single instance that crosses the 80% utilization, or if the average utilization in a given zone crosses above 80%? The former would trigger an alert for every single VM, leading to possibly hundreds of separate alerts. The latter will lead to one alert for a widespread issue. To consolidate the number of alerts, add a “group by” to your alert policy; see step 3.d in the documentation. 

Suppose there is a spike in 500 errors for a few seconds. In this case, the anomaly may be worth looking into — but that is less severe than when error rates persist for several minutes. Setting policies based on duration window helps indicate impact over time. From there, if you want a “warning” alert policy for a slight duration of 1 minute, and a “critical” alert policy for anything longer than 5 minutes, you can create two separate policies, each with the defined duration window, and apply different severity levels to each one.

Sometimes, an alert serves as a leading indicator, an early warning for a situation that would be much more complicated to act on once it arrives. For example, in monitoring the quota consumption of a cloud service, it’s necessary to know well in advance if a service is approaching maximum usage so that a customer can manually request a limit increase, which can take time to approve and cannot be automated.

If an issue is upcoming or ongoing, it needs direct intervention, whereas an issue that has already happened, it may require investigation. Thinking in advance about the urgency of intervention helps to set the right thresholds for prioritization, and can avoid too many small noisy issues from overtaking more pressing ones.

Who should I notify, and how?

Even in the most robust, automated system, it may still be necessary to send alerts to people. In those situations, the more context people have about an issue, the better. When creating an alert policy, you can use the Documentation section to add your internal playbooks and Google Cloud landing pages, and other links for your on-caller to begin an investigation, and add relevant labels for context.

Once you have identified the people who need to know about an issue, you can decide how to notify them. In addition to using notification channels, Google Cloud offers Webhooks to send notifications to any public endpoint and Pub/Sub for any private endpoint for third party integrations. integrations with 3rd party tools. 

For example, if the problem is urgent, you may need to notify a person directly, using SMS or a third-party paging tool. If the issue is important but less urgent, you can use a third-party tool to create a ticket that enters a queue. While using different channels may seem painfully simple, a common problem is using only one notification channel regardless of the urgency or severity level. If a team gets urgent notifications all the time, even for minor issues, they may get stressed out or desensitized, which in turn distracts them from the priority. 

If alerts go to a low-priority channel, like a group email (which tends to be the default for cloud quota consumption), people may only see them after it’s too late. Only notify people who need to know that there is a problem, and leave it to their discretion to inform others. It may take some planning, but the goal is to avoid spamming people with irrelevant notifications. Different channels enable prioritization and appropriate visibility.

If the on-call person is the only person who can resolve an urgent issue, you should page them. There are plenty of urgent situations where the person on-call can’t handle the resolution on their own, so paging them is just the first step in an urgent escalation. (See a separate discussion and examples of incident managementhere.) If the issue isn’t critical, the alert should generate a ticket in the queue. The on-call person can still work on it–just not as an “interrupt.”

Concluding Guidelines

From establishing the relevance and intended outcomes of alerts, we can use quick rules of thumb to help us get the most out of alerts by keeping a high signal-to-noise ratio.

What to alert on:

Alerts should be actionable for Ops teams and relevant to end users.

For a mature Ops team, move to alerting on SLOs

To get started with Monitoring, see the Out of the Box Packages of alerts, dashboards, and metrics on the Integrations Page.

Alerts are the “start” of an action or an investigation; they may only represent a small portion of what you monitor.

Confirm the intended granularity of your alert policy. At what level do you consider a system impacted? Per violating Instance or on the average of all instances? 

When to trigger an alert:

Determine if the alert is a leading or lagging indicator of a problem. Consider that impact changes over time.

Pick thresholds that reflect key points of change. Are my users experiencing something bad past this point? Is there enough time for my team to intervene?

If needed, divide thresholds into different severity levels for the same indicator. Pick thresholds based on assumptions about impact, response time needed, and how much risk is mitigated and accepted (e.g. error budgets for SLOs).

Who to alert, and how:

Only notify people you intend to act in response, and trust them to inform more people if needed.

Add playbooks and labels in the Documentation section to help your on-call team start their investigation

Using a specific notification channel or third-party tool, directly notify the person on-call if the situation requires an immediate response.

For less urgent issues, use a different channel or a third-party ticketing system instead of a direct notification to mitigate alert fatigue.

Get Started Today

To get started with alerting in your project, visit Alerting in the Google Cloud Console. 

For introductory guides on how to set up alerts, see: 

* Alerts:Uptime Checks to Pub/Sub Topics

* Alerts:Log-Based Errors to Pub/Sub Topics

If the guidelines in this article helped you create effective alerts or you have any questions, please join us at the nextReliability Engineering Discussion.

Data scientists and engineers need large-scale compute resources but often need more know-how or the engineering team to create and manage the underlying infrastructure.  Dataproc Serverless, the industry’s first autoscaling serverless product for Spark, simplifies their experience by not having to think about infrastructure for their work.

Dataproc Serverless uses Spark properties to determine the compute, memory, and disk resources to allocate to your batch workload. These property settings can affect workload quota consumption and cost (see Dataproc Serverless quotas and Dataproc Serverless pricing for more information).

This blog post will demonstrate how you can use these Spark properties to meet the runtime and cost service level agreements of your Spark jobs.

Before you begin

This blog post assumes the following:

You have an Apache Spark jar available for Dataproc Serverless.

Your project has the appropriate quota for testing.

You have no insight into what amount of cpus/memory/disk your job requires.

You’ve exhausted code optimization opportunities.

You have tried other methods such as optimizing GCS file structures / file types etc.

Dataproc Serverless and Apache Spark tuning is an iterative process. This guide outlines the steps to fine-tune your spark properties to meet cost and performance requirements. It will require you to submit your job several times.

Dataproc Serverless pricing

Dataproc Serverless for Spark pricing is based on the number of Data Compute Units (DCUs) and the amount of shuffle storage used. DCUs and shuffle storage are billed per second, with a 1 minute minimum charge.

For more information, visit: Dataproc Serverless pricing

Spark Persistent History Server and GCS Logging Output

Set up a Spark Persistent History Server (PHS) for your jobs.  This will help during debugging and troubleshooting.  All executors and driver logs are available in Cloud Logging while the Spark job is running.  Also, Spark applications are visible in PHS while running (PHS > Incomplete Applications).  Executor and driver logs are available in Cloud Logging, but the PHS has the Spark event logs that provide insight into Spark app execution (DAG, executor events, etc).  The logs are actually stored in GCS; however, the UI to access them is available through the PHS (when the PHS is not running, Spark won’t send the logs to GCS).

Additionally, set up a Google Cloud Storage bucket and pass these properties to your Spark job.

Tracking job performance/cost progress

Throughout your iterative experimentation, keep track of each job’s properties and the impact on runtime and cost.

Submitting your job

If your job runtime / DCU usage is satisfactory, you can stop here. This can be found on the details tab of the Dataproc Serverless batch console:

As your job runs, monitor the maximum-needed value in the Monitoring tab of the Dataproc Serverless batch console:

In the above example: maximum-needed = 36.5, but the job was only allocated with 1 executor.

The maximum-needed value is computed by adding the count of running tasks and pending tasks, and dividing by the tasks per executor.  Dataproc Serverless will attempt to meet this demand based on the autoscaling-related properties passed during job submission.  These properties are discussed in the next section.

Fine-tuning your Spark properties

Allocated job resources

For sake of simplicity, below are three example configurations for varying workloads. Experiment with each configuration to determine which best suits your job. To view the default Spark property values and learn more visit: Spark properties | Dataproc Serverless Documentation | Google Cloud

Autoscaling

Autoscaling helps cut costs by only using required resources, but may affect runtime and stability of the job. It is best to leave this as default until you are confident that runtime or job stability do not meet your needs.

Several properties influence scale range and rate, elaborated below. To view the default Autoscaling property values and learn more visit: Dataproc Serverless for Spark autoscaling.

Range

Configuring a range will provide Dataproc Serverless jobs with a minimum and maximum number of executors.  You can also set an initial amount (default is 2).  Setting a sensible initial amount of executor instances will reduce runtime (less time spent autoscaling). Limit costs by setting maximum number of executors.

Rate

When autoscaling, Dataproc Serverless jobs compare the current number of executors vs. the maximum-needed (total number of running tasks and pending tasks, divided by the tasks per executor) and will increase/decrease the current number of executors to reach equilibrium. The rate of increase/decrease is determined by executorAllocationRatio and decommission.maxRatio.

The executorAllocationRatio property determines how quickly the job scales UP. The decommission.maxRatio property determines how quickly the job scales DOWN. 

Decommissioning too many executors at the same time with shuffle or rdd migration could severely hurt performance of shuffle fetch. Therefore it is best to set this to a low value, 0.1-0.3 range. This will reduce performance regression caused by decommission. 

For sake of simplicity, here are three example configurations for autoscaling rate:

No Autoscaling

Since we know how many executors your job needs based on the maximum-needed value, and since autoscaling takes time to scale up, we can set the min and max executors to be identical and turn off autoscaling. This will give your job the exact number of instances beginning / during your job run. Because dynamicAllocation is still enabled, you’ll be able to see the monitoring tab in the Dataproc Serverless job console.

Compute and disk tiers

One more way to boost the performance of your Dataproc Serverless job is to utilize compute and disk tiers. By default, your jobs will be set to standard tier. If you want higher per-core performance (compute) or better IOPS and throughput (disk), consider setting these properties to premium tier. Be aware that jobs will be billed at a higher rate with premium tiers.

Putting it all together

Consider the above options when submitting your job. Some example scenarios:

1. A job requires medium allocated job resources and moderate autoscaling?

2. A job requires maximum allocated job resources, fast autoscaling, and we know the initial executor amount to be 50?

3. A small-sized job has a predictable and consistent resource requirement of 20 executors and a need for high IOPS?

Common errors explained

Keep an eye on your project quota. If autoscaling is enabled and the project quota limit is hit, your job will not receive the resources it requires and may fail with this error.

Disk Usage = number_instances * number_cores * disk_size

CPU Usage = number_instances * number_cores

CourseGrainedSchedulerError

This error usually relates to insufficient resources. Try increasing the driver cores/memory and/or the number of executor instances.

FetchFailedException

Try setting spark.reducer.fetchMigratedShuffle.enabled=true

For more information, visit Spark Dynamic Allocation Issues and Solutions 

Next steps

As discussed in this blog post, Dataproc Serverless uses Spark properties to determine the compute, memory, and disk resources to allocate to your batch workload. These property settings can affect workload quota consumption and cost (see Dataproc Serverless quotas and Dataproc Serverless pricing for more information). Thus, it is essential to understand how these properties function and interact with each other.  

To get started with Dataproc Serverless, check out our Serverless Spark Workshop or the Dataproc Serverless Quickstart. 

If you want to learn more, check out some related blog posts:

Simplify data processing and data science jobs with Spark on Google Cloud 

Spark jobs that autoscale and made seamless for all data users

Running Persistent History Servers

Backup and DR data is a valuable business asset, and ensuring that it’s safe and accessible is essential. In particular, you want to be able to monitor your backups to ensure that the data is indeed protected and that you can quickly recover it in the event of a disaster, user error, or failed upgrade.

Users of Google Cloud’s Backup and DR service often ask:

What are the best practices for monitoring my backup and recovery jobs?

How can I proactively identify issues and troubleshoot? 

How can I create alerts and be notified about important events? 

The good news is that Google Cloud Backup and DR now integrates with Cloud Logging and Cloud Monitoring tools. Now, you can monitor backup events, jobs, appliance health, resource consumption and user actions in the same way you monitor other Google Cloud workloads and services. 

With this integration, you can now:

Monitor events related to backup and restore – With detailed Google Cloud Backup and DR service event logs in Cloud Logging, you can now use log based alerts to create a customisable notification, in near-real time. This lets you monitor important events such as  backup job failures, jobs that did not run, local backup storage saturation, and network failures.

Configure fine grained alerting – You can write custom event queries on a wide variety of dimensions such as event severity, application type, application name, job type, job name, error message and many more. This gives a lot of flexibility to define alerts for specific events or conditions within a system, thus reducing noise and helping you identify and fix issues easily. 

Get notified on your preferred notification channel – Google Cloud offers seven pre-built notification channels that let you receive customisable notifications directly over email, SMS, Slack or the Google Cloud mobile app. Alternatively, you can integrate with your own monitoring and event management tools using webhooks or Pub/Sub. 

Derive useful insights to troubleshoot issues – With Log Analytics, you can use BigQuery to query your data using SQL queries and generate operational insights, which can help you reduce time spent troubleshooting. For example, you can analyze the key reasons that backup and restore jobs fail for a given application or application type. 

The integration also lets you proactively discover deeper issues related to backup and recovery, such as capacity-related issues or certain job failures that happen periodically, by monitoring recurring events in the logs over time. With log-based metrics, you can: 

Create trend charts to track important metrics such as the number of backup or restore jobs that failed during a day/week/month 

Receive a notification when the number of occurrences crosses a threshold, for example receive a notification if a snapshot pool saturates more than five times a week 

Monitor trends in data, such as latency values in logs, and receive a notification if the values change in an unacceptable way.

Getting started

The time to learn that your backup failed should never be when you go to restore. By integrating our Backup and DR service with Cloud Monitoring and Cloud Logging, you can get valuable assurances about the health of your backup and business continuity processes with the same tools that you use to manage other Google Cloud workloads. To get started, check out the Backup and DR event logs page. You can also watch a video that shows you how to set up and use the service and configure some custom alerts.

When I was growing up in Thailand, I witnessed the challenges facing farmers including rising food demand, shortage of labor, and uneven crop yields caused by climate change. As a result, many smallholder farmers found themselves trapped in a vicious circle, unable to reduce food insecurity due to low yields, but lacking the resources to invest for a more profitable future.

I was determined to make a difference. After earning my master’s degree in Information Management I joined a research project with the University of Tokyo where we used sensors to monitor spinach fields in Thailand. The results of this early experiment in precision farming were impressive. We proved that it was possible to grow organic crops with minimal use of fertilizers, while consumers benefited from higher quality spinach grown in Thailand.  

This inspired me to foundListenField in 2017. Our mission is to transform farm management by collecting data from multiple sources, including field sensors, soil scanning, weather data, seasonal forecasts, and satellite imagery. By modeling this data, we provide farmers with insights that enable them to optimize production ‘from soil to harvest’.

A bumper crop of farming data

Artificial intelligence and machine learning play a central role in our prediction platform, combining crop health monitoring, growth prediction, and soil nutrition analysis. Farmers can apply real-time insights to their schedule from ourFarmAI Mobile App, while ourFarmAI Dashboard enables agri-food businesses to collaborate with agronomists and farmers so that all parties benefit from higher profit margins and more sustainable growing strategies.

Another important feature of our business model isAgroAPI, which makes our analytics available to third parties. Clients can embed deep analytics in their applications, including crop growth prediction and remote sensing analysis, without needing to develop complicated algorithms and data pipelines by themselves.

We are also excited about our research into genomic prediction in collaboration with the Japanese government and several research companies. Using ourData-Driven Breeding Platform, breeders and seed companies can upload their genomic data and gain practical insights that help accelerate the reproduction of high-quality seeds and plants.

Today, more than 30,000 farmers use our technology especially in Vietnam and Thailand where it is used to improve rice, cassava, and sugar cane yields. The technology is also being rolled out for orange and mango farmers enabling them to monitor individual trees and adjust irrigation to improve the sweetness of the fruit at harvest time.

Responding fast to changing conditions

We were using another cloud provider to run our business, but one of the ListenField team members drew our attention to Google Cloud. As well as the technology, we were also attracted bythe Google for Startups Cloud Program which provides us with Google Cloud credits that cover our first and second years of Google Cloud usage. We also met our Account Representative, who provides us with training, business and tech support, and Google-wide discounts. It’s great to have a point of contact that can help us on our startup journey and make the most of Google’s resources.

By reducing the pressure on our finances and human resources, we were able to experiment and adapt in response to early experiments. This flexibility also enabled us to demonstrate a compelling business case to new and existing investors.

Our Google Cloud Platform Partner,Navagis, gave us additional momentum thanks to their expertise in mapping and geospatial data. They also played a crucial role in the integration ofGoogle Earth Engine, which we use to map agricultural areas.

We also useFirebase for application development andGoogle Workspace for team collaboration.Colab andVertex AI enable us to build, deploy, and scale our machine learning models quickly, ensuring that we remain competitive and attractive to new customers.

Giving female entrepreneurs the opportunity to flourish

Both the Google Cloud team and our colleagues at Navagis helped us to navigate the challenges many early-stage startups face. My background is in science and academia, so I appreciated the business mindset offered by both organizations to help us continue to grow and scale.

Being a female entrepreneur leading a startup can also be tough, but Google Cloud and Navagis helped me to build a strong network, access funding, and make my voice heard. Today, ListenField has several female executives, while 50% of our researchers and many of the farmers on our platform are women.  

Above all, Google Cloud helps us to power an agriculture revolution in south-east Asia. Smallholder farmers can transition from analog to digital farming, improving their yields and reducing waste. The benefits to the economy are also significant including greater food security and reducing the use of industrial fertilizers that generate potent greenhouse gasses.

And that’s just the beginning of what we can do. Our next milestone is to reach 50,000 farmers and cut one million tonnes of greenhouse gas emissions. It sounds ambitious, but with the Google Cloud and Navagis teams behind us, I’m confident that we will reach these targets.

If you want to learn more about how Google Cloud can help your startup, visit our page here to get more information about our program, and sign up for our communications to get a look at our community activities, digital events, special offers, and more.

When our founders began Mambu in 2011, their goal was to bring the latest digital technologies to the banking and finance world. Banking, in particular, is an industry built on decades of deep legacy technology. So initially, Mambu was embraced by microfinance — 100 organizations in 26 countries in just the first two years. 

Since then, acceptance of modernizing core banking services by using composable, cloud technologies has grown across financial services institutions (FSIs). We now service top-tier banks, fintech startups, and other finance organizations across six continents, helping them deliver flexible, personalized, customer-centric banking products and services that their customers can depend upon.

One of the main reasons we’ve been able to scale the Mambu composable banking platform across the globe and at our current pace is our partnership with Google Cloud. The decision to move forward with Google Cloud happened for several reasons.

1. Flexibility and openness. Many FSIs are on hybrid and multicloud technology stacks, as they may still be transitioning from legacy systems, or have data residency requirements that have led them to use different clouds in different regions. Mambu meets customers wherever they are in their cloud journeys. We support interoperability without vendor lock-in. This need for openness, as well as the scalability benefits, led Mambu to evolve our platform on Google Kubernetes Engine (GKE). Many customers use open-source Kubernetes because, this common foundation can help streamline integration, speed up time to market, and reduce development. Just as important, the Google Cloud open cloud approach matches our company values. 

2. Security and data residency. For customers in highly regulated finance industries, security isn’t just top of mind,  it’s the No. 1 requirement. In addition to Google Cloud’s secure infrastructure, external audit certifications, and encryption, its wide array of regions has allowed us to expand into more countries, where we serve banks that must meet local data residency requirements. For example, Google Cloud’s Jakarta Cloud Region, has allowed us to support Bank Jago in Indonesia as it brings more financial inclusion to the unbanked in that country.

3. Availability. It’s critical for banks to maintain basic financial functionality, like accepting deposits and serving cash, even amid a service disruption. We needed a cloud partner with impeccable redundancy, failover, and disaster recovery capabilities.

In addition to GKE, the Mambu platform uses several other Google Cloud services for specific functions, including Cloud Armor, Cloud Load Balancing, Cloud VPN, Cloud Memorystore, and Google Cloud Operations.

Another important reason we chose to partner with Google Cloud was its expansive ecosystem and commitment to innovation. We are midway into a three-year journey to modernize our own technology stack to meet customer needs.

Building a roadmap with Google Cloud

Our customers need a core banking technology platform that will grow with them as they bring to market innovative services built on the latest technology advances. For Mambu to be that platform, we need a cloud partner that supports and scales with our growth. While we originally built our cloud architecture on GKE and Compute Engine (among a few other Google Cloud services), we’re now looking to a serverless future where we can scale more easily and leverage managed services within Google Cloud and its partner ecosystem to focus on our core offerings. 

These are just some of the modernization and customer-led innovations that we’re cooking up: 

More workloads in GKE: Like many companies, our cloud transformation is a work in progress. While much of our codebase is in GKE, we’re continuing to break up some larger pieces of code into microservices to increase agility and velocity, enabling us to make consistent updates to discrete areas of our platform without affecting the whole. GKE is the leader for orchestrating microservices at scale and continues to be a natural fit.

Native BigQuery integration: Mambu customers collect a tremendous amount of data within the platform that can be used for analytics, personalization, and other use cases. We’re planning to create a seamless integration for feeding core banking data from Mambu into BigQuery so that customers can better leverage their valuable data.

CloudSQL vs. self-managed MySQL: We have almost completed migrating from our own MySQL instances to managed Cloud SQL databases, which will open up new opportunities to implement customer-centric solutions such as BigQuery integration.

A serverless future with Cloud Run: Compute Engine is working well for Mambu, providing the flexibility to choose the virtual machines that best balance performance and cost needs. As we seek even more time and cost efficiencies, we believe the elastic scalability of a serverless architecture built on Cloud Run will get us there, and we’re considering going serverless in the future. Doing so would abstract infrastructure for simpler management, while allowing us to fire up containers to meet our customers’ high transactions-per-second needs, spin them down when not needed, and pay only when they run. It would also boost security: Without long-running compute, there are no patches or fixes, and each new instance is isolated and fresh by default.

These are just a handful of examples of the ways we want to best leverage Google Cloud services to simplify how we manage our tech stack, as well as continue to bolster security, scalability, and performance. There are many other ideas we’re exploring: using Dataproc and Datastream to support the specific data needs of Islamic banking, Cloud Functions so that customers can run their own queries against Mambu, and AI-enabled features. 

At Mambu, our mission is to empower our customers to deliver great modern financial experiences easily to everyone around the world. Every time Google Cloud opens a new data center, we can enter a new market. Every time we move to a new-to-us managed service via the Google Cloud Marketplace, we free up time to build new ways to deliver customer-centric banking solutions. And so, we look forward to continuing this partnership with Google Cloud well into the future.

Verve Group is an ecosystem of demand and supply technologies fusing data, media, and technology to deliver results and growth to both advertisers and publishers – no matter the screen or location, no matter who, what, or where a customer is. Classifying massive amounts of this unstructured data at scale is the first step in helping to surface relevant, high-quality content to users—and that’s where natural language processing (NLP) comes in.

Verve Group uses the NLP API from Google Cloud’s Vertex AI to fetch data for their internal content classification quality verification and as an additional source for building categorization models. By leveraging the NLP API’s Content Classification models, which are now generally available and offer Google’s latest large language model (LLM) technology, Verve Group powers classification through an updated and expanded training data set with over 1,000 labels and support for 11 languages (Chinese, French, German, Italian, Japanese, Korean, Portuguese, Russia, Spanish, and Dutch join previously-available English). 

Verve Group has been using Google Cloud’s NLP API since day one, because of both the ease of implementation and the quality compared to competing NLP products. With documentation that is “comprehensive and self-explanatory,” the NLP API “allows for fast adoption and implementation, from test models all the way to production,” said Rami Alanko, GM of Verve Group.

Leveraging the NLP API has facilitated Verve Group’s fast go-to-market motions by enabling its customers to quickly discover and classify new content. “Operating on a global level with tens of different regions and languages, we have still been able to maintain high quality for our product and high retention rates with our clients,” Rami shared. “In a recent client case, we achieved 82% improvement in CTR when optimized with content quality measurements enabled by the API. In another client case, we drove brand safety risk down to 0.16% from 4% thanks to classification quality. Along with the new functionalities of the Google NLP, I can only see this trend continuing to strengthen.”

Verve Group is excited to further expand their NLP use cases by leveraging the new Content Classification models, which have already helped them expand their classification inventory, improve the quality and performance of their quality verification for customers, and unlock new use cases for NLP. “Our classification model accuracy improved 41% using Google NLP as a verification partner,” said Rami.  

Additionally, Verve Group is now using the API for metadata analysis on a  large image database. “We browse the database and run the image metadata via our classification. This flow enables us to classify images reliably aligned with our standard classification.  We pretty much use the same data flow for our runtime in-app textual content analysis, therefore allowing for close to real-time consumer engagement,” Rami added.

To learn more about how companies are leveraging NLP API from Google Cloud Vertex AI, click here, and to learn more about Google Cloud’s work with foundation models and generative AI, read The Prompt on Transform with Google Cloud.

Could enhancing the developer experience be the key to setting your business apart? Mckinsey research found that nearly 70% of the top economic performers are leveraging software for a competitive edge. Given that your software is a crucial differentiator, it’s essential to concentrate on the critical resources producing this software — your developers, their happiness and their experience. Google Cloud is on a mission to help every organization digitally transform their business and industry, and we’re putting developers at the center of this transformation.

Developer challenges, today

In the last decade, several trends such as “shift-left,” API-first, containers, microservices, and an explosion of choices from open-source software have transformed the developer experience. While these changes accelerate value creation for businesses, they also create new friction points and challenges.

These challenges often manifest in the following scenarios:

Getting started with a new cloud can be daunting and requires significant cognitive investment. Quickly, you’ll be facing a maze of disparate cloud services and APIs. 

As the complexity of building applications grows, developers rely upon disparate, and sometimes contradictory, knowledge sources. These range from product documentation, to product collateral, best practice information, third-party sites and other developer community information. Developers must synthesize knowledge from these disparate information silos, which requires constant context switching. 

The shift-left approach extends the developer’s responsibilities to include testing, security, performance and UX assessments.

Laying the groundwork for developer productivity and development velocity

Indicative of its importance, there exists a plethora of recent industry, academic, and UX research focusing on improving developer productivity and overall development velocity. From an organizational perspective, leaders and the team topologies they create, influence the culture, process, and clarity of communication that set the tone for developer experience. Google has consistently focused on working with developers to improve development velocity and has built trust with development leaders by exporting and evangelizing the methodologies, structures, and technologies that we use to make our own teams successful. We’re not only promoting the adoption of efficient and effective practices, but also empowering our customers to innovate, scale, and thrive within their industries. For example: 

Conducting primary developer research and reporting on findings with DORA

Creating and evangelizing our internal best practices like Site Reliability Engineering (SRE)

Contributions to security standards like SBOM and SLSA

Putting the right structures and practices in place for your teams is only the beginning of the journey. In the face of today’s challenging environment, developers require a renewed focus on how the platform and tools you’re already paying for will help them overcome the aforementioned friction points and challenges. 

At Google Cloud, we are intensifying our efforts to innovate the platform and tooling options for your developers. Our approach begins with completely reimagining the developer experience and delivering capabilities to make development fast, easy, and secure. This reimagined experience impacts every aspect of the developer journey. Commencing before you become a Google Cloud customer, our new experience is interwoven through onboarding, inner-loop development and outer-loop development so that you can continuously deliver customer value. 

The Google Cloud developer experience

We’re committing to a “modern developer experience” that empowers developers to: 

Stay in “flow state” longer: Developers are fully immersed in their favorite development activities and maintain context and focus. Whether they’re learning about Google Cloud services, authoring or accessing APIs, writing, testing, or debugging code, they should be able to perform most of their duties from within their IDE to minimize interruptions.

Shorten “feedback loops”: We aim to prevent distractions during the iterative processes that developers perform hundreds of times during the development cycle. Whether it’s validating a local code change or component integrations, or ensuring that the features they built meet customer expectations, optimized wait times during these feedback loops have a significant impact on developer effectiveness.

Reduce cognitive load by “shifting down”: Developers can leverage platform abstractions to automate and push more tasks into the platform. Taking full advantage of managed services reduces the mental effort on your team, and allows them to focus on value-generating tasks.

This new developer experience is brought to life by product capabilities that reimagine how you:

Discover and learn about Google Cloud

Onboard developers to Google Cloud

Improve inner-loop development

Secure outer-loop development

Continuously deliver value to customers

Discover and learn Google Cloud 

Our developer experience journey begins even before developers become Google Cloud customers. We understand the complexities of choosing the right cloud provider, and the myriad of technical and organizational challenges to navigate. Traditionally, organizations research information about cloud services, APIs, features, costs, and best practices from multiple information sources, performing time-consuming trial and error exercises. We aim to streamline this journey by “shifting-down” onto the Google Cloud platform. This process begins with developers asking a simple question in Google Search. For example, “How do I perform AI/ML image processing on Cloud Functions?” 

The developer is presented with a streamlined onramp that prioritizes their learning, while accelerating their workload deployment. After deployment, developers navigate through interactive code modifications, iterative re-deployment workflows embodying Google Cloud best practices. Our workflows are designed to meet developers where they are by offering seamless integration between the Google Cloud console and the toolchains and workflows of their choice. Ultimately, the developer goes from idea, to search, to experiencing an “aha moment” in less than an hour. The process establishes a solid technical foundation to build on (i.e., deployed infrastructure, reusable Infrastructure-as-Code, and example application source code), an understanding of the technical next steps to achieve their goals, and confidence in Google Cloud.

Onboard developers to Google Cloud

When onboarding new developers, it can take weeks or even months before they become productive. Cloud Workstations, built upon Google Cloud’s opinionated security best practices, not only enhances your security posture with a centrally managed IDE but also mitigates exfiltration by preventing local storage of code. As a cloud-based, pre-configured, and centrally managed development environment, Cloud Workstations reduces the time required to provision and configure a secure development environment. From the beginning, Cloud Workstations simplifies onboarding for new and remote developers, while increasing developer productivity with fast IDE access via a web browser from anywhere. Staying true to Google Cloud’s commitment to open source and flexibility, Cloud Workstations also supports popular IDEs so your developers can work wherever they code best. To support our shift-down methodology, we are adding more capabilities to Cloud Workstations. These include:

Enhanced enterprise security and data loss prevention: Improved integration with BeyondCorp provides additional safeguards to ensure the safety of your code.

Extended IDE support: We now support Posit Workbench with Rstudio so that developers who build R-based applications can benefit from all the capabilities of Cloud Workstations.

Expanded infrastructure support: Cloud Workstations instances can now run on GPUs for better performance during the development and testing of AI/ML applications.

Improve inner loop development

Duet AI reimagines the developer experience with an industry-first embedded AI assistance for Google Cloud users at all manner of skill levels, across the surfaces they use to interact with Google Cloud. It uses Gen AI to foster a more intuitive experience to build, run, and manage cloud applications, data, and infrastructure services. With Duet AI natural language chat assistance, interacting with the cloud is as easy as talking with a friend, whether that’s in the console, Code editor, Cloud Workstations or Cloud Shell.

Developers can elevate their productivity by harnessing the power of gen AI to create new, enhanced experiences, all while staying engaged in a “flow state” to learn, understand, evaluate, design and implement a variety of technologies without leaving your IDE:

Stuck on how to complete a task? Just ask Duet AI. 

 Want to delve deeper into a specific topic? Request more information from Duet AI.

Need a quick high-level overview of a topic? Ask Duet AI for a summary. Duet AI will provide you with a concise summary of the topic.

Want to decide which service to use for your use case? Ask Duet AI to compare capabilities of GCP services.

Duet AI serves as a pair-programming partner and can assist with a variety of design and coding tasks for frontend, backend, fullstack and SQL developers. From designing APIs, to authoring, editing, and debugging code, developers are using Duet AI to streamline their processes and enhance their productivity through the focus-encouraged ‘flow state’ allowing them to optimize both time and effort. Duet AI also helps shorten “feedback loops” by automatically generating unit tests for you to enable faster code validation. With Duet AI’s assistance capabilities, you can:

Generate APIs: Duet AI can automatically generate APIs that are consistent with your existing APIs and that comply with your enterprise policies. Simply specify your needs in natural language, and continue refining your APIs through the same conversational interaction.

Complete your code: Duet AI can suggest the next line of code while you’re authoring code in your IDE, including Visual Studio Code, IntelliJ, or the BQ Editor.

Generate code: Based on natural language instructions, Duet AI can generate high-quality code, saving you time and effort, particularly on complex projects.

Explain code: Duet AI can explain code written by you or others, so you can understand how it works and what it’s doing. This can be helpful for debugging or when you’re trying to learn a new language or framework.

Suggest and generate unit tests: Duet AI can propose unit tests for your code, ensuring you achieve maximum code coverage. This can help you catch bugs early on and avoid problems in production. 

Build complete low-code apps conversationally: For AppSheet users, Duet AI enables anyone to create complete low-code apps via natural language.

Contextually code aware: Duet AI’s deep integration with Google Cloud allows it to leverage APIs and native services to provide relevant code suggestions. With support for 20 programming languages (and growing,) Duet AI is the versatile coding companion for your entire development team.  

Duet AI also offers invaluable assistance into IP compliance risks when automatically suggesting and generating code for you. It can provide code license attribution if the suggested code falls under restrictive license, and provides the necessary context so you can make an informed choice on whether to use that code. Should the code generated require a permissive license, it provides you with the information needed to attain license attribution.

Secure outer-loop development

Today, enterprise customers are under pressure to enhance the security posture of their software supply chain while minimizing the overhead incurred when meeting regulatory obligations around their software supply chain. The shift-down mindset helps ease a developer’s cognitive load by enabling the security in the underlying platform, and strengthening software supply-chain security throughout the development lifecycle. By leveraging the totality of the platform, you can improve the security of your application, such as ensuring the security of your application’s open-source dependencies with our Assured Open Source software. The platform provides a trusted source for you to access Google curated and tested OSS packages. 

Then, to offer developers more choice and meet them where they are, we’ve expanded our DevSecOps partnership with GitLab. Combining the power of Google Cloud S3C with GitLab’s DevSecOps platform to generate a secure supply chain with end-to-end visibility. Our new partnership will empower users to leverage a seamless connection between the two when it comes to deploying software securely. This integrated application development solution will provide best-in-class reliability, efficiency and end-to-end security. In Phase 1, the integration of Artifact Registry into the GitLab SaaS offering’s workflow will enable customers to easily set up secure pipelines in GitLab that automatically connect builds to Artifact Registry and to GKE, providing a high performance, secure delivery of containers to Google Cloud with full regionalization and compliance with DRZ requirements.

Continuously deliver value to customers

The adoption of microservices and CI/CD undeniably entrusts development teams with the freedom to work independently, generate value, and accelerate development velocity. However, it’s critical to remember that these microservices are components of a larger and more intricate value chain that ultimately benefits the end user. That value chain involves integration with other microservices, SaaS services, and enterprise services. This end-to-end integration that surfaces the business flows to the end customer is the conduit of value for your customers. While there has been a lot of focus on optimizing micro “feedback loops” during the development cycle, it’s equally critical to optimize the value-delivery feedback loop so that you can learn and improve based on your customers’ feedback.

To enable this continuous value delivery, developers employ a variety of application integration techniques. These integrations require specialized knowledge of third-party apps, application integration tools, and skills to deal with ongoing maintenance of these integrations. To accelerate this continuous value delivery, integration work is also shifted down into the platform and simplified using Duet AI.

With 75+ pre-built connectors that can easily help you connect Google Cloud services like BigQuery and Pub/Sub with third-party applications like Salesforce, MongoDB, Oracle and SAP.

With Duet AI, we simplify application integration even further. You simply provide your requirements in a natural language prompt and you get a list of integration flows to choose from. Duet AI uses your enterprise context, based on which it will automatically generate flows, documentation and test cases using your existing APIs and assets helping you accelerate continuous value delivery.

The faster you can deliver this value to your end-customers, the more you can accelerate the customer-feedback loop. Whether you are delivering prototypes, preview features, or doing experiments to choose between designs, you can accelerate the delivery of your overall application to your end-users and shorten your customer feedback loop.

2023 is already shaping up to be one of the most consequential years in technology. Gen AI is redefining work across sectors, enhancing the quality, scale, and innovation of developers. Duet AI, interwoven throughout Google Cloud, allows developers of all skills to collaborate with their AI companion via the intuitive, conversational interface. Yet, the full benefits of these game-changing technologies are only realized when working in tandem with focused developers who feel support from their leadership. The shift-down methodology is more than automation checklists and managed services, it is a commitment to the health and psychological well-being of the team underpinning your entire digital infrastructure. Minimizing developer burden not only rejuvenates your teams, but refocuses their attention on innovation and value creation. To learn more about how “shifting down” and Duet AI can drive development velocity, sign up for the Duet AI early access. Embrace the change, accelerate your progress, and unlock the true potential of your team with shift-down and Duet AI.

Moving from manual to automated document processing has been one of the great success stories of AI, removing countless hours of repetitive, error-prone manual labor, and freeing people up to focus on delivering real value. Google Cloud’s Document AI has been a key part of this success. 

But new ways of taking in documents can expose organizations to various levels of risk and create new opportunities for fraud to occur, particularly in high-risk industries like financial services. Know Your Customer (KYC) and anti-money laundering (AML) regulations require these institutions to collect and process documents to understand their clients better. Automating those processes without adequate controls can instead increase their risk.

The risks for FinTechs are in particular very high, as introducing operational efficiencies like automation into their organization is usually a key organizational goal — but it also creates opportunities for various types of nefarious activities. A good example is organized criminals creating fake accounts which can lead to fraud and money laundering activities. This is why fraud resiliency is one of the biggest challenges for FinTech Product and Operational teams, particularly in the automation of customer onboarding.

Manual verification is incapable of identifying most forgeries in digital documents since they leave little-to-no visible traces. After processing tens of millions of documents, Resistant AI found that up to 17% of digital bank statements used for loan applications, account openings, and other purposes worldwide have been tampered with, and 15% of company registration certificates submitted worldwide when opening corporate bank accounts are fake. In addition, serial fraud attacks which rely on easy-to-find document templates provide a cheap way to scale attacks beyond human controls and create armies of throwaway accounts that can be used as synthetic money mules. 

That’s why Resistant AI developed Document Forensics: an easy-to-deploy API that can automate document fraud checks to secure any workflow built with Google Cloud’s Document AI. 

Automated document fraud checks

The core goal of Resistant AI’s Document Forensics is to provide a quick decision to accept, deny, or escalate a digital document for additional review. The submitted document is qualified, classified, and verified within seconds to speed up the review process, whether in PDF or image format, to ensure a seamless and easy experience.

Quality: Can you read the document? This step checks the documents to ensure they are of sufficient quality to be processed for Optical Character Recognition (OCR), and aren’t hiding fraud behind camera flashes, low resolution, image blurs or more. Causes of rejection can be used as feedback for customers to submit a new version. 

Classification: Is it the right kind of document? This step ensures you only take in documents that meet your acceptance criteria. You shouldn’t be taking in and processing ID cards if all you’re looking for are utility bills.

Verification: Should you trust it? This step checks the document over 500 ways for signs of tampering. After individual analysis, it will compare all submitted documents — everything from metadata to the color temperature and scene composition — to find patterns of document reuse, template farms, and mass fraud attempts across accounts. 

Together, these three steps make sure human intervention isn’t wasted on clear fraud cases which should be declined, or on documents that show no signs of tampering and which should be automatically processed by Google Cloud’s Document AI extraction. When a document is escalated for review, it is displayed in a user interface (UI) which makes every single finding  that determined the verdict available to the reviewer so they spend less time making a decision.

Payoneer, a major international merchant services provider, was able to reduce manual document fraud reviews to just 18% of their intake, and their customer approval representatives agreed with the verdicts 99.2% of the time. In addition they’re now stopping hundreds of serial fraud attempts per day at onboarding. 

Habito, the UK’s leading digital mortgage broker, was able to detect 32% in incremental fraud compared to their already existing fraud solutions, and reduce fraud investigations by 52 minutes per case.

How to bring Resistant AI Document Forensics and Google Cloud Document AI together

By operating in the same cloud, customers can expect to fully leverage the security and privacy standards they’ve come to expect from Google Cloud to apply to Resistant AI — speeding up procurement processes. 

Document AI also goes beyond simply extracting data from documents, but enriches the result with the Google Knowledge Graph to check company names, addresses, phone numbers, and other details against entities on the internet.  

Learn more about Resistant AI on Google Cloud here and check out their listing on Google Cloud Marketplace.

Being a data-driven organization is vital in today’s business landscape. Data-driven insights drive innovation, identify growth opportunities, and ensure competitive advantage. But to realize these advantages, organizations must make sure they’re using high-quality data — data that’s timely and reliable, free of errors, omissions, and duplicates. Business intelligence and decision-making that runs on low-quality data can have serious implications for customers and for the business. 

Data observability — a set of techniques and real-time processes that reveal whether your data contains signals that need investigating — helps ensure that your data is accurate, complete, and current, allowing your organization to make informed business decisions with high confidence. But data observability that ensures data accuracy and validity in a timely manner, particularly on large volumes of data, requires distributed computing frameworks and large-scale processing power — making Google Cloud the architecture of choice to run its data observability solution.

Data quality challenges of the modern data stack

Organizations have traditionally taken a rules-based, manual approach to ensuring data quality. But this classic approach [Figure 1] no longer meets the demands of data teams due to four key limitations:

Operational overhead: Managing rules manually is a time-consuming and inherently error-prone process. What’s more, it’s a reactive process, so data teams discover the need for a new rule only when something goes wrong.

Cost-effective scalability: As data volume and complexity grow, it becomes more difficult to manage data quality and accuracy manually, particularly when that data is spread across multiple data storage systems. Without automation built into your processes, scaling rules-based management requires additional human and computing resources, which pushes costs higher. 

Batch and streaming data, including semi-structured data: Building data quality at scale means analyzing, discovering, and detecting issues across hundreds of tables and across all (or most) attributes within these sources, which often contain semi-structured JSON data. These types of workloads are computationally heavy. Often, a single data pipeline consists of multiple underlying source systems like data warehouses, Delta Lakes, analytic databases, and streaming systems — all of which make a rules-based approach difficult to execute and scale.

Incomplete visibility: Rules-based data quality management assesses data against predefined metrics and alerts to issues like inconsistencies, discrepancies, and inaccuracies. But anomalies, outliers, and drifts in data can go undetected without data observability tools. These tools use machine learning and statistical analyses to learn from your data, predict issues, find root causes, and monitor changes in your data for signals worthy of investigation.  

Google Cloud: A platform of choice for Telmai’s Data Observability Platform

Built and run on Google Cloud, Telmai Data Observability Platform helps organizations monitor and manage the quality of their data by providing a centralized view of data across all data sources. Telmai’s engine performs data profiling and analysis to identify potential issues, such as missing values, duplicate records, and incorrect data types; ML-based anomaly detection to surface unexpected values in data that may indicate problems and to predict what can be reasonably expected; and continuous monitoring to detect changes in data quality over time.

To build an advanced, AI-driven architecture [Figure 2] capable of scaling and solving data quality and observability challenges, Telmai considered various cloud computing platforms. The company selected Google Cloud because Google Cloud’s managed and serverless components — including Dataproc, Pub/Sub, Google Kubernetes Engine (GKE), and BigQuery — offered the best economic and performance value. With a modern architecture on Google Cloud, Telmai’s observability solution is quick to deploy for startups and scale to enterprise use cases on demand.

By using Spark (running via Google Cloud Dataproc) as the processing engine, Telmai is able to decouple data quality analysis and anomaly detection from the underlying operational systems such as BigQuery, which offers three advantages:

Open architecture: Telmai’s decoupled architecture enables calculating (and continuously monitoring) data quality metrics and thresholds for any kind of underlying system in customers’ data pipelines without overloading those systems. This open architecture gives data architects the flexibility to add, upgrade, or swap data systems without worrying about redesigning their data quality and observability. 

Scalability: Spark gives Telmai the ability to design highly optimized and scalable algorithms specific for data monitoring, where the traditional approach of running SQL queries fell short. A scalable architecture with elastic resources gives Telmai Data Observability Platform the ability to monitor hundreds of millions of data metrics and their trends so the solution can validate data efficiently, in parallel, and with high throughput — for example, data with 100 million+ rows of JSON structures of 1,000+ attributes. Relying on Dataproc service provides resilience and ease of operation at scale; clusters autoscale, spin up, and stop when required. This keeps solution costs down for customers.

Security and operations: Google Cloud provides a high level of control for security. Security ranges from basic-level measures like multi-factor authentication, single sign-on, key store, granular roles, and ease of separating development from production environments, to more sophisticated measures such as Security Command Center to track and remediate vulnerabilities. 

One of the biggest security risks for any organization is human error, so it’s critical to define and operate infrastructure as code to standardize deployments and minimize chances of misconfigurations or exposures. By leveraging Kubernetes and GKE, Telmai is able to not only further reduce such risk, but also to run Telmai deployments with low effort in customers’ accounts entirely (private cloud option), ensuring that all data stays within their safe perimeter.

These benefits together gave Telmai tremendous speed in development, including supporting new integrations. By relying on managed services like Dataproc and GKE, Telmai can focus on building its application rather than  on operating the software. The infrastructure, auto-scaling, security, DevOps, and more are readily available for Telmai via the Google Cloud stack.

Google Cloud and Telmai: Partners in data observability

Google Cloud is helping technology companies like Telmai build innovative applications on Google’s data cloud with simplified access to technology, helpful and dedicated engineering support, and joint go-to-market programs.  

Learn more about Google Cloud’s open and innovative generative AI partner ecosystem and about Telmai and Google Cloud.

DNS is a foundational component of Internet-delivered applications. When DNS outages occur they cause service outages and can even cause issues with monitoring and remediation of problems. This is exacerbated with multi-tenant setups in Cloud where services from many providers are impacted since they all share the underlying DNS infrastructure. While a single-provider deployment is the most common, easy to set up and maintain, it’s also a single point of failure. Utilizing two different providers for hosting public DNS improves the overall reliability.

One way to set up multi-provider DNS is using zone transfer capability supported within the DNS protocol (RFC 5936). However, most Cloud providers don’t support this capability. A second way is to ensure that your DNS updates are reflected in both providers, typically done by tweaking the CI/CD pipelines to create and update DNS records in both providers at the same time. Managing and maintaining multiple providers tends to be operationally heavy and prone to human errors and this is one of the many reasons why multi-provider setups are not as common as they should be. 

What if there’s a simpler way to achieve this? You continue to update a single provider but there’s an over-the-top solution that automatically picks up the changes and reflects it onto another provider. We are pleased to announce the launch of Terraform scripts that make it easy to use Cloud DNS as a second authoritative DNS server for public DNS hosting. Our solution relies on automation built using Terraform and OctoDNS to monitor your current DNS zones and reflect those changes in a managed DNS zone hosted on Cloud DNS. The scripts ensure that new Managed zones are created in Cloud DNS and when individual DNS records are updated, it automatically transfers the updates to the corresponding DNS zone in Cloud DNS. You don’t need to make any changes to your existing CI/CD pipeline. Finally, we’ve launched this solution in open source to enable you to customize it to your needs and environment. The rest of this post provides two approaches to consider depending on your needs.

Dual-provider in an active-active setup

The first option we’d like to discuss is the use of two providers in an active-active setup. Since both providers are primary, you need to configure the appropriate DNS name servers of both providers as the responsible name servers for your domain. This may involve working with the registrar to publish updated “NS records” in the parent domain registry, as well as publishing matching NS records at the domain’s “zone apex”. This enables resolvers to fall back to the remaining available provider’s nameservers in case one of the providers is down. If the outage is not resolved quickly, NS records can be updated to delist the affected provider’s nameservers until the problem is resolved. Resolvers typically keep track of the health status of the nameservers they use, and partial nameserver downtime is typically not service-impacting.

Pros

No single point of failure – the probability of both providers having simultaneous outages is very small

Failover is transparent and automatic

Cons

Need synchronization between the Primary and Secondary — more complex than a single-provider deployment

Potentially higher cost since the same service is being provided from two different providers

Dual-provider in an active-passive setup

A second way to set this up is with an active-passive configuration. You could designate one provider as primary and another as secondary. You list only the primary vendor’s nameservers in the “NS records” given to your domain registrar and all DNS requests by default will be sent to the primary. 

Behind the scenes, all DNS Zone updates are sent to both the primary and the secondary providers. Choosing two cloud providers makes this easy because of API support — you can call both the APIs back to back to ensure the primary and the secondary zones are in sync. You can also utilize open-source tools like OctoDNS to keep the providers in sync. 

The active-passive setup requires a trigger to be set up that detects failures and switches to the secondary DNS service. The trigger could become the single point of failure so the design and deployment of the trigger becomes critical. Note that some parent (TLD) domains have long NS TTLs, and recovery for domains registered under a public suffix can be slow in active-passive mode.

Pros

More reliable than a single provider system — it’s unlikely that both providers will have simultaneous outages 

Potentially lower cost since the secondary is only storing the zone information and does not serve query traffic as long as it is in secondary mode

Cons

Need synchronization between the Primary and Secondary — more complex than a single provider deployment

Down time until secondary provider is switched to be the primary, and for some time after as resolver caches catch up

Need a triggering mechanism to switch between primary and secondary services — could become the SPOF

Use Cloud DNS as a second authoritative provider

We believe that the active-active configuration is the best mechanism to minimize downtime due to an outage. In order to assist customers to use Cloud DNS as a second authoritative provider, we’ve launched Terraform scripts that do the bulk of the automation work. 

Our solution helps you to do a one-time data transfer or do periodic transfers of DNS records over to Cloud DNS depending on your needs. The caveats to this is that any provider-specific DNS records (like custom apex CNAME records) will not work on Cloud DNS. We also don’t support DNSSEC for multi-provider zones. Currently, we support migrations to Cloud DNS from Amazon Route 53 and Azure DNS. This solution can be easily extended to other API-enabled providers. 

You can find the open-sourced version of these scripts at Google Cloud’s Github repository.

The investment management business is more data-driven than ever. The Portfolio Management Group (PMG) at BlackRock — the world’s largest financial asset management company — is responsible for a wide range of investment strategies, using data at all stages in the process to make investment decisions, manage multi-asset portfolio risk, and keep track of portfolio performance. As data used for the investment process continues to grow in volume and variety, data operations become increasingly complex and challenging, requiring a variety of skills and tools to manage. 

To adapt to the rapidly changing asset management environment, data operations teams need to improve the efficiency, accuracy, and flexibility of their data processing through customizable data pipelines to extract and organize information from a growing number of disparate data sources. These include traditional financial datasets and alternative datasets. By making data more consumable, the data operations teams can empower investment and research teams — i.e., data consumers — to make better decisions at the speed of business.

BlackRock’s Data Strategy & Solutions team set out to build a custom solution for data operations by harnessing the power of Google Cloud to automate data profiling and to identify  data quality issues early, such as missing values, duplicate records, and other outliers. The solution uses a unified architecture that can calibrate to the evolving data landscape in the asset management industry. This in turn improves the quality, speed, and usability of the data used by the investment and research teams.

Architecting the solution on Google Cloud

BlackRock built a platform architecture using BigQuery as the data fabric to support a standard model for all data across internal and external sources, along with seamless integration with other services including Cloud Storage, Pub/Sub, and Cloud Functions, to automate data pipelines that proactively identify data errors and inconsistencies. The following illustration shows the architecture at a high level.

Without this unified solution, the data operations team would have to check feeds, databases, application logs, and other data sources and then manually reconcile inconsistencies that are susceptible to human error. That would make alert prioritization more difficult and maintenance more of a burden. However, the solution unifies all sources in a single location and stores them in BigQuery as a common data model with standardized extensible data schemas. Data pipelines with customized alert algorithms run through scheduled queries, Cloud Functions, and Compute Engine, which allows the data to be prioritized and acted on much faster through automated data flows between data managers and data consumers. 

As a result, the source applications remain unchanged. Platform engineers have maximum flexibility to create bespoke algorithms for alerts as long as the algorithm output matches the standard data model. As shown in the diagram above, the data operations team has a unified experience to identify potential data issues and resolve them through a common interface at speed and at scale. 

Principles of the platform include:

Simplicity and scalability – BigQuery serverless architecture eliminates the overhead with predictable query performance at scale (hundreds of millions of records) with native support for real-time and batch processes to identify and resolve data quality issues in real time.

Extensibility – Stand up new data pipelines and workflows with ease to enable new data sources through native integration between Cloud Storage, Pub/Sub, Cloud Functions and BigQuery.

Future-proofing – Self-optimizing BigQuery offers a fully integrated intelligent data fabric with support for structured, unstructured, and spatial data with “built-in” AI/ML capabilities, which unifies the distributed data to help automate data management and power analytics at scale.

Realizing the benefits 

Every month, BlackRock’s Data Strategy & Solutions team processes over 30,000 datasets, while more than 6,000 alerts are actioned to provide consistent, high-quality data to the company’s investment teams. This enables them to  be more agile in responding to market changes.

The unified solution helps prioritize access across six data sources and 15 unique alert algorithms, allowing urgent issues to be addressed 30% faster by presenting the decision-making information through a consistent interface. 

Improved data operations processes have reduced the time it takes to add new data sources and alerts from days to hours. This represents an 80% reduction in onboarding time. As a result, data engineers can have an additional 20 hours a month to focus on other strategic tasks, such as developing new data products and services.

Our data organization successfully implemented DataOps best practices to adapt to the changing data landscape, reduce the time to value, and minimize the total cost of ownership.  

By creating a unified solution on Google Cloud, the BlackRock Data Strategy & Solutions team can scale data consumption by providing a reliable, scalable, and secure platform for storing, processing, and analyzing data. The data processing and operations architecture can integrate simply with rapidly developing Google Cloud capabilities, including generative AI technology, and it will continue to transform how our investment teams interact with data to identify opportunities, assess risk, and make better investment decisions.  

For more information on how Google Cloud is benefitting the capital markets industry, visit this page.

If you want to get the latest news on city-sponsored events, apply for a building permit, or pay for a parking ticket in Italy, you will likely interact withGruppo Maggioli. The 100-year-old, family-managed company provides software solutions to more than 5,000 Italian municipalities, as well as other organizations.

In recent years, the company expanded operations through acquisitions in Latin America and across Europe. Today, Gruppo Maggioli’s software division is known for its website and mobile app platforms that bring higher quality, reliability, speed, and transparency to public administration.

Better services for citizens and businesses

Gruppo Maggioli began exploring cloud technologies to unlock the power of SaaS business models and more seamless infrastructure management. The company ultimately chose Google Cloud to support its digital transformation.

“We standardized on Google Cloud for its high reliability, scalability, and performance to ensure that we could modernize scalable public administration applications and infrastructure,” said Robert Ridolfi, director, IT Service and Technologies Business Unit, Gruppo Maggioli.

The company worked withInjenia, a Google Cloud partner that Gruppo Maggioli purchased in 2021, to help accomplish its cloud migration goals. Injenia has Google Cloud specializations for application development, work transformation, and machine learning (ML).

Injenia experts trained Gruppo Maggioli’s IT staff and helped transform application architecture from monolithic to microservices usingGoogle Kubernetes Engine (GKE).

“We implemented continuous integration and automation using GKE Autopilot mode to flexibly and efficiently manage our resources while accelerating deployments and our time to market for new features,” said Franco Cucchi, software security specialist and cloud architect, Gruppo Maggioli. “We also see greater collaboration between our product engineering team and platform engineering team.”

Gruppo Maggioli integratedCloud SQL to manage its databases. In addition, the company’s teams useFilestore as a fully managed, low-latency NFS solution for its applications running in GKE. As its needs for additional Google Cloud products evolve, the IT team makes purchases through Google Cloud Marketplace to save time in procurement with the convenience and visibility of unified billing.

Cloud security and data sovereignty

Gruppo Maggioli deployedCloud Armor to bolster its security posture through ML-based capabilities trained on its applications to provide adaptive protection against attacks on applications and websites. The IT team is also testing use ofSecurity Command Center and Chronicle Security Operations for threat detection, investigation, and response.

Data governance was a key factor in Gruppo Maggioli’s selection of Google Cloud given GDPR requirements and additional local data regulations across Europe. Google Cloud’s robust global meshed network of data centers, including 11 European regions and more than 30 edge Points of Presence (POPs) spanning Europe, supports compliance requirements that require data residency while providing fast performance.

“Google Cloud’s POPs in Rome, Milan, and now Turin help us simultaneously deliver great experiences for our clients across Italy and address their data governance concerns,” Cucchi said.

Cucchi noted that Google Cloud helps Gruppo Maggioli respond to increasing client interest in hybrid cloud solutions. His team is exploring use ofAnthos to manage GKE clusters and workloads running across Google Cloud, on-premises, or other public clouds.

“Our business is driven by constantly changing customer needs,” said Cucchi. “With Google Cloud, we’ve established a solid reference implementation and a trusted, preferred cloud provider relationship. At the same time, we appreciate the opportunity for great flexibility with a consistent managed Kubernetes experience across environments that seamlessly adapts to our customers’ requirements.”

A brighter future

So far, Gruppo Maggioli has migrated to Google Cloud several of its popular applications, including  Municipium, Sicraweb, Icaro, Sportello Telematico, and others. “We see benefits in terms of high quality of service (QoS) and availability. This helps us deliver excellent services to our clients,” said Cucchi.

Gruppo Maggioli usesLooker Studio to analyze its business data and create impactful visualizations of key metrics. With timely insights, the company’s leadership can make better decisions to guide its future.

The firm’s digital transformation has included organizational change. “We’ve helped our IT staff develop new skills in cloud best practices to maintain our legacy of innovation,” said Stefano Palazzi, software factory director, Gruppo Maggioli. “As a result, Gruppo Maggioli continues to modernize and improve integrated digital public administration services to benefit citizens, businesses, and other constituents.”

This week we hosted Google Cloud Next in San Francisco! It was great to be back together in-person with the Google Cloud community and ecosystem. Highlights included the opening keynote, packed with breakthrough product announcements and customer stories, which you can also watch in just 13 minutes.  Also check out the fun and inspirational developer keynote, complete with an original song and cool demos. We hosted 250+ educational breakout sessions, and thousands of developers were buzzing around our Innovators Hive. On top of that, we loved having partners integrated throughout Next, from the show floor to the sessions, to the evening parties throughout the week.

Generative AI was a big focus throughout, particularly the many new offerings across Vertex AI and Duet AI, highlighting all the new ways to cloud.

Too much happened this week to share it all here, but we’ve pulled together a summary of all of the announcements so you can feel like you were there. Read on for the complete list of 161 (!) announcements. 

Data and AI Cloud

AI platforms and tools

1. Duet AI in Google Cloud is now in preview with new capabilities, and general availability coming later this year.

2. Vertex AI Search and Conversation, formerly Enterprise Search on Generative AI App Builder and Conversational AI on Generative AI App Builder, are both generally available to make it fast and easy to build generative chatbots and custom search engines.

3. New multi-turn search in Vertex AI Search supports follow-up questions without starting the interaction over. 

In addition, we built out the Vertex AI platform:

4. We added new models to Vertex AI Model Garden including Meta’s Llama 2 and Code Llama and Technology Innovation Institute’s Falcon LLM, and pre-announced Anthropic’s Claude 2. 

5. The PaLM 2 foundation model now supports 38 languages, and 32,000-token context windows that make it possible to process long documents in prompts.

6. The Codey chat and code generation model offers up to a 25% quality improvement in major supported languages for code generation and code chat. 

7. The Imagen image generation model features improved visual appeal, image editing, captioning, a new tuning feature to align images to guidelines with 10 or fewer samples, and visual questions and answering, as well as digital watermarking functionality powered by Google DeepMind SynthID. 

8. Adapter tuning in Vertex AI is generally available for PaLM 2 for text.

9. Reinforcement Learning with Human Feedback (RLHF) is now in public preview. 

10. New Vertex AI Extensions let models take actions and retrieve specific information in real time and act on behalf of users across Google and third-party applications like Datastax, MongoDB and Redis.

11. New Vertex AI data connectors help ingest data from enterprise and third-party applications like Salesforce, Confluence, and JIRA.

12. Grounding in Vertex AI roots generative outputs in your enterprise data, to increase confidence in your generative AI search and conversational applications. 

13. Vertex AI now supports Ray, an open-source unified compute framework to scale AI and Python workloads. 

14. We announced Colab Enterprise, a managed service in public preview that combines the ease-of-use of Google’s Colab notebooks with enterprise-level security and compliance support capabilities. 

15. Next month, we’ll make Med-PaLM 2, our medically-tuned version of PaLM 2, available as a preview to more customersin the healthcare and life sciences industry.

And to enhance MLOps for generative AI, we announced:

16. Automatic Metrics in Vertex AI lets you evaluate a model based on a defined task and “ground truth” dataset.

17. Automatic Side by Side in Vertex AI uses a large model to evaluate the output of multiple models being tested, helping to augment human evaluation at scale. 

18. A new generation of Vertex AI Feature Store, now built on BigQuery, helps you avoid data duplication and preserve data access policies. 

Data analytics

We announced a wealth of features across our data analytics offerings:

19. Duet AI is in preview across a variety of products in the data analytics portfolio such as Looker, BigQuery, and Dataplex. 

20. BigQuery Studio, in preview, is a single interface for data engineering, analytics, and predictive analysis to simplify end-to-end data workflows.

21. We announced enhanced support for open source formats like Hudi and Delta Lake within BigLake, and added performance acceleration for Apache Iceberg. 

22. BigLake can now be your single lakehouse, with cross-cloud materialized views and cross-cloud joins in BigQuery Omni.

23. Spark integration on Google Distributed Cloud extends the power of fast analytical query processings to on-premises to support data residency requirements. 

24. New governance capabilities in Dataplex for data lineage, quality, and metadata management help users understand what data to analyze, and train ML models on trusted data sources to help improve accuracy. 

25. BigQuery data clean rooms can help you understand your Google and YouTube campaign performance.

26. Now you can access Vertex AI foundation models, including PaLM 2, directly from BigQuery. 

27. New model inference in BigQuery lets you run model inferences across formats like TensorFlow, ONNX, and XGBoost, and new capabilities for real-time inference can identify patterns and automatically generate alerts. 

28. For model tuning, we added vector and semantic search in BigQuery. 

29. You can automatically synchronize vector embeddings in BigQuery with Vertex AI Feature Store for model grounding. 

30. Now you can access thousands of datasets from hundreds of providers including Acxiom, Bloomberg, Equifax, Nielsen, and Zoominfo directly in BigQuery. 

Databases

Transactional and operational databases are the lifeblood of the organization. 

31. AlloyDB AI is available in preview via downloadable AlloyDB Omni to help you build enterprise gen AI applications everywhere. 

32. AlloyDB Omni, the downloadable edition that runs on Google Cloud, AWS, Azure, on premises, and on your laptop, is moving from technology preview to public preview. 

33. Duet AI in Database Migration Service provides AI-assisted code conversion to automate the last mile conversion of Oracle database code to PostgreSQL. Sign up for the preview.

34. Duet AI in Cloud Spanner is in preview to help with generating code to structure, modify, or query your data using natural language. 

35. Support for Oracle in Database Migration Service with Cloud SQL for PostgreSQL as the target is now GA.

36. Cloud Spanner Data Boost, now GA, lets you to analyze your Spanner data via services such as BigQuery, Spark on Dataproc, or Dataflow with virtually no impact on your transactional workloads.

37. Support for auto-generated keys in Cloud Spanner allows schema authors to push down critical identity logic into the database. 

38. A new BigQuery Export to Bigtable Reverse ETL feature in preview lets you serve analytical insights from your applications without having to touch any ETL tools. 

39. The fully managed Memorystore for Redis Cluster is available in preview. This is an easy-to-use, open-source compatible Redis Cluster service that provides up to 60 times more throughput than Memorystore for Redis, with microseconds latencies.

40. The new Google Cloud Ready for Cloud SQL program recognizes partner solutions that have met integration requirements with Cloud SQL.

41. The Bigtable change streams feature allows you to capture data changes to a Bigtable table as the changes happen, letting you stream them for processing or analysis, and is now GA.

42. Cloud Bigtable request priorities, in private preview, let you execute large workloads that are not time-sensitive e.g. analytical queries, as low priority jobs on a Bigtable cluster, thus minimizing the impact of batch processing on serving workloads. 

43. You can now create a copy of a Cloud Bigtable backup and store it in any project or region where you have a Bigtable instance, and retain your backups for up to 90 days.

Business Intelligence

We extended our Looker business intelligence ecosystem and showed that the future of Looker is powered by AI.

44. Duet AI in Looker is now in preview. 

45. The Looker semantic layer is now open to Tableau in preview, and to Microsoft Power BI in general availability.

46. Looker Studio users now have native access to the Alteryx Designer Cloud for data preparation, and enhanced cloud connectivity, starting with Microsoft Excel and CSV files from storage platforms including Sharepoint and OneDrive.

47. An integration with Looker partner Sisu Data will be generally available later this quarter, helping customers determine the root cause of changes, spot data outliers, and identify next steps for analysis.

Dev Cloud

Google Cloud’s Dev Cloud covers all our infrastructure — compute, network, storage, regions, etc. — as well as platforms and developer tools.

Compute

We talked about chips that are optimized for AI workloads:

48. A3 VMs, based on NVIDIA H100 GPUs and delivered as a GPU supercomputer, will be generally available next month.

49. The new Google Cloud TPU v5e, in preview, has up to 2x higher training performance per dollar and up to 2.5x inference performance per dollar for LLMs and generative AI models compared to Cloud TPU v4. 

50. New Multislice technology in preview lets you scale AI models beyond the boundaries of physical TPU pods, with tens of thousands of Cloud TPU v5e or TPU v4 chips. 

51. Support for Cloud TPUs in GKE is now available for Cloud TPU v5e and Cloud TPU v4. 

52. Support for AI inference on Cloud TPUs is in preview.

We enhanced our general-purpose Compute Engine families with new models and features:

53. The Arm-based C3A, powered by AmpereOne processors will be in preview next month. 

54. A preview of AMD-based C3D VMs is coming next month, and will offer the largest instance sizes in our general-purpose portfolio, with up to 360 vCPUs and 2.8TB of DDR5 memory. 

55. We unveiled Titanium, a system of purpose-built, custom silicon and multiple tiers of offloads that enables offerings like Hyperdisk and our 3rd generation C3 VMs.

56. C3 VMs support for Hyperdisk Extreme is now in preview and supports 500K IOPS.

57. The SSD-based Hyperdisk Balanced is now in preview, with up to 2x the performance of previous generation Persistent Disk SSD. 

58. Hyperdisk Storage Pools is now in preview, allowing customers to provision capacity and performance in aggregate Pools, and then thinly provision Hyperdisk volumes from those pools.

59. Our VMs have new uptime SLAs: a 99.95% uptime SLA for memory-optimized VMs, and 99.9% (up from 99.5%) for all other VM families. 

60. Future reservations, now in preview, is a new Compute Engine feature that allows you to reserve compute capacity for a future date. 

61. C3 is now certified for SAP.

62. Workload Manager has new capabilities for securing and deploying SAP workloads, in preview.

63. In preview in October, we’ll offer a new VMware Engine node with 2TB RAM and options from 64 – 128 vCPUs and up to 50TB storage for Google Cloud VMware Engine, as well as three new storage options: NetApp Volumes, Filestore, and Storage-only (VSAN) nodes. 

Networking

64. We announced Cross-Cloud Network, an open and programmable global cloud networking platform. 

65. Private Service Connect now supports 20+ different Google and partner managed services. 

66. The Cloud Firewall Plus tier is in preview, a cloud-first next-generation firewall, or NGFW.

67. The new Network Service Integration Manager simplifies the setup and operation of partner NGFWs from Checkpoint, Cisco, Fortinet, and Palo Alto Networks. 

68. VPC Spokes support in Network Connectivity Center in preview lets you scale VPC connectivity, providing reachability between a large number of VPC spokes. 

69. The internal Application Load Balancer now supports global access, which allows private clients from any Google Cloud region to access internal load balancers residing in any other Google Cloud region, and global backends, which allow internal Application Load Balancers to health-check and send traffic to globally distributed backend services. 

70. New Service Extensions callouts for Cloud Load Balancers let you customize services such as specialized monitoring, logging, traffic steering, or authentication.

71. A new automation solution toolkit for the Google Cloud Load Balancers global frontend lets you integrate and automate products including Cloud Armor, Cloud Load Balancing, and Cloud CDN into popular CI/CD platforms. 

72. Cloud Application Load Balancers now support cross-project service referencing. 

73. New mTLS client-side authentication is available for global external Application Load Balancers.

74. Auto-deployment for Cloud Armor Adaptive Protection is now generally available.

75. Palo Alto Networks is integrating its Prisma Access natively in Google Cloud.

76. Broadcom is integrating its Secure Web Gateway natively in Google Cloud. 

77. The Titanium Network Adapter, part of our Titanium system of offloads and available in A3, C3, and H3 VMs, delivers up to twice the throughput and three times the packet processing speed of prior VM generations.

Hybrid and multi-cloud

We added capabilities and features to our Google Distributed Cloud family, GDC Edge and GDC Hosted:

78. GDC Hosted now offers pre-trained models for speech, translation, optical character recognition (OCR), and Workbench. 

79. Vertex Prediction and Vertex Pipelines are coming to GDC Hosted in preview in Q2 2024, as is Document Translation API service. 

80. Database Service for GDC Hosted will now support AlloyDB Omni as a new managed database engine in preview. 

81. GDC Edge will support Dataproc in preview in Q4 2023. 

82. The new GDC hardware stack features the 4th Gen Intel® Xeon® Scalable Processors and high-performance network fabrics with up to 400 Gbps throughput. 

83. We introduced new hardware configurations for GDC Edge that are ruggedized and optimized for retail stores and restaurants.

84. GDC Hosted now offers support for new guest operating systems, customer-managed VM images, a package repository for VM runtime configuration, simplified networking, backup and restore tooling, a CLI and streamlined APIs. 

85. The new GKE Enterprise will be included in all Google Distributed Cloud deployments at no additional cost.

86. ElasticSearch, MongoDB Enterprise Advanced, and SAP on Google Distributed Cloud are available through Google Cloud Marketplace.

87. Boundary proxy on GDC Edge in preview provides visibility and auditability by inspecting and logging all management traffic between Google Cloud and GDC Edge (in preview). 

88. New Bastion host on GDC Edge, in preview, enables you to easily view and control Google Cloud’s access to GDC Edge for troubleshooting purposes, supporting you with operational sovereignty and compliance requirements.

89. Now you can automatically create and provide the resource mapping for all your managed keys with key tracking on GDC Hosted, in preview. 

90. BYO-certificate on GDC Hosted enables you to upload certificates issued by a third-party certificate authority, in preview.

91. With new survivability mode on GDC Edge, you can continue on-prem operations on GDC Edge even if you lose connectivity to Google Cloud, now in preview. 

Platforms

92. GKE Enterprise, a new premium edition of our flagship container orchestration platform, lets platform teams can manage large Kubernetes deployments across multiple clusters and multiple teams around the world, all from a single management console. 

93. GKE now supports Cloud TPU v5e, A3 VMs with NVIDIA H100 GPUs, and Google Cloud Storage FUSE on GKE (GA). 

94. Duet AI in GKE (preview) provides generative AI assistance specifically trained on GKE documentation to help platform teams cut down on time it takes to learn and operate Kubernetes. 

Developer experience

We took our commitment to an exceptional developer experience up a notch:

95. Jump Start Solutions application and infrastructure solutions, are now GA.

96. Application Integration — a no-code integration platform as a service (iPaaS) designed to empower you to weave together your applications — is now GA. 

97. We partnered with GitLab to offer a secure DevOps solution with integrated source management, artifact management, CI/CD, and enhanced security features. 

98. Duet AI is available console-wide, including in GKE, Cloud Run, and Cloud operations suite. It will also be available in Apigee API Management and Application Integration in preview for trusted testers here.

Migration and Management

99. We’ve revamped (re-RaMPed?) our Rapid Migration Program, including the new Cloud Capability Assessment (CCA) for RaMP online assessment and a detailed new migration-planning dashboard.

100. Migration Center, a unified migration service within Google Cloud console, is now GA, and includes the new SAP Cost Estimator.

101. Google Cloud VMware Engine is now available in 18 regions, most recently in Turin, Italy (europe-wst12), Santiago (southamerica-west1), and Delhi (asia-south2). 

102. The new flexible ve2 node platform powered by 3rd Generation Intel® Xeon® Scalable Processors (formerly named Ice Lake) and next-generation VMware Engine networking are now available for Google Cloud VMware Engine.

103. Event-driven transfer in Storage Transfer Service is now GA. 

104. Storage Transfer Service now offers support for transferring data from on-premises Hadoop Distributed File Systems (HDFS) sources to Cloud Storage, currently available to trusted testers. 

105. SAP Security Validation, now available in public preview for Google Cloud Workload Manager, provides SAP-specific infrastructure and application-level security checks. 

106. Workload Manager’s Guided Deployment Automation service is now available in public preview and generates custom automation code to modernize to S/4HANA.

Geo and sustainability

107. The Google Maps Platform team introduced a suite of Environment APIs for solar, air quality, and pollen.

Security Cloud

108. Duet AI in Mandiant Threat Intelligence in preview helps surface the most prevalent tactics, techniques, and procedures (TTPs) used by threat actors against organizations by summarizing our frontline threat intelligence.

109. Duet AI in Chronicle Security Operations in preview aids in threat detection, investigation, and response for cyber defenders by simplifying search, complex data analysis, and threat detection engineering. 

110. Duet AI in Security Command Center in preview offers near-instant analysis of security findings and possible attack paths.

111. Mandiant Hunt for Chronicle, now in preview, provides continual threat hunting by Mandiant experts on Chronicle data. 

112. Agentless vulnerability scanning, powered by Tenable, has been integrated into Security Command Center to detect operating system, software, and network vulnerabilities on Compute Engine virtual machines, now in preview.

113. Security Command Center now allows organizations to design their own customized posture findings (GA) and threat detectors (in preview).

114. Confidential Computing running on 4th Gen Intel Xeon Scalable CPUs with TDX technology is now in private preview.

115. We expanded the coverage footprint of our Sensitive Data Protection offerings with enhanced integration for Dataplex and Dialogflow (now GA), and in preview for Cloud SQL.

116. Assured Workloads Japan Regions is now in preview, offering customers controlled environments that enforce data residency in our Japanese regions, options for local control of encryption keys, and administrative access transparency.

117. The official Google Cloud Certified Professional Cloud Security Engineer Exam Guide is now available. 

Collaboration Cloud: Google Workspace

118. Duet AI for Google Workspace is generally available, and you can get started with a no-cost trial. 

119. Duet AI for Google Workspace can create a whole new presentation in Slides, complete with text, charts, and images, based on your relevant content in Drive and Gmail. 

120. Duet AI in Google Meet helps you look and sound your best with new AI-powered enhancements, including studio look, studio lighting, and studio sound, as well as dynamic tiles and automatic face detection so remote attendees can see everyone in a meeting room, with each in-person attendee getting their own video tile with their name.

121. Duet AI in Google Meet can capture notes, action items, and video snippets in real time with the “take notes for me” feature, and send a recap to attendees after the meeting. It can even catch latecomers up with “summary so far,” or attend a meeting in your place with “attend for me.”

122. We’re enhancing smart reply in Gmail with Duet AI, allowing you to draft longer personalized replies with a single tap.

123. The enhanced Google Chat experience is available with powerful new features, including Duet AI in Google Chat. To help you keep up with messaging, we’re bringing direct messages and spaces together into a unified conversation list, with a chronological home view, @-mentions, and starred conversations, with intelligent prioritization of your messages, based on your communication patterns, coming early next year.

124. Google Chat will soon support up to 500,000 participants in a single space, to help build thriving communities, even in the largest organizations.

125. Huddles in Chat are a new way for teams to communicate in real time, using quick-to-join audio and video conversations, powered by Google Meet. With huddles, instead of jumping out of the conversation into a meeting, the meeting integrates directly and smoothly into the Chat experience. 

126. More third-party and Google Workspace apps are supported in Chat, including an updated Google Drive app that lets you respond to comments and sharing requests, as well as new support for Workday, Loom, Zoho, and LumApps. Mio will provide support for message interoperability with other major platforms.

127. In the lead up to Next, we announced new capabilities, including those powered by AI, to help prevent cyber threats, provide safer work with built-in zero trust controls, and better support for our customers’ digital sovereignty and compliance needs.

Customers 

128. We announced the winners of our Google Cloud Customer Awards.

129. AdoreMe, a direct-to-consumer brand, uses generative AI in Google Workspace to improve customer experiences, create production-worthy marketing materials, and speed up innovation.

130. Bayer Pharmaceuticals is working with Google Cloud to use gen AI, including testing Med-PaLM 2. 

131. Canoo, a leading advanced mobility company, will deploy a range of Google Cloud’s AI, data management and security technologies to maximize the value of the data coming from its electric vehicles.

132. CAPCOM is harnessing Google Cloud’s infrastructure to support its global player base for AAA game launches like Street Fighter 6.

133. Dun & Bradstreet is collaborating with Google Cloud on gen AI to drive innovation across multiple applications. 

134. The Government of El Salvador announced a multi-year deal with Google Cloud to digitally transform the country with Google Distributed Cloud (GDC) Edge, as well as to establish an office presence and Center of Excellence (CoE) in the country.

135. The Estée Lauder Companies Inc. (ELC) and Google Cloud announced an expansion of their strategic partnership to pioneer new uses of gen AI for online consumers.

136. Fox Sports will use Google Cloud’s gen AI capabilities to quickly search footage from more than 1.9 million videos and create custom video content in near-real-time to delight and engage its audiences.

137. GE Appliances (GEA) is expanding its partnership with Google Cloud to enhance and personalize consumer experiences, like recipe creation and appliance maintenance, with gen AI.

138. General Motors and Google Cloud will share new details on how the two companies are collaborating to bring conversational AI technology into millions of GM vehicles.

139. Ginkgo Bioworks, which is building the leading platform for cell programming and biosecurity, and Google Cloud announced a five-year strategic cloud and AI partnership, intended to enable Ginkgo to develop and deploy AI tools for biology and biosecurity. 

140. Hackensack Meridian Health will leverage Google Cloud’s gen AI technologies to develop and deploy solutions that will help clinical staff to focus more on care, improve overall decision-making, and better personalize the patient experience.

141. HCA Healthcare and Google Cloud will improve workflows on time-consuming tasks such as clinical documentation so physicians and nurses can focus more on patient care with gen AI. 

142. Huma will use Google Cloud’s gen AI to enhance its regulated disease management platform.

143. Infinitus is using Google Cloud’s gen AI to streamline provider-payor interactions, simplify resource-intensive operations, and improve response times. 

144. Meditech is working with Google Cloud to empower employees, and ultimately better serve its healthcare provider customers with gen AI, including Med-PaLM2. 

145. MSCI is expanding its partnership with Google Cloud to accelerate the development of gen AI solutions for the financial services industry. 

146. Runway has chosen Google Cloud to build, accelerate, and manage model deployment and bring gen AI to more creatives.

147. Six Flags will use Google Cloud’s gen AI to create an in-park app that helps improve visitor experience and park operations in real time. 

Startups and Public Sector

148. Eleven more leading generative AI startups have chosen Google Cloud to develop gen AI and bring their products to market.

149. The new Google for Startups Startup School: AI is a six-week in-depth online AI training course that will launch in January 2024. 

150. A new VC AMA ebook, Startup Advice from VCs: 10 Lessons from the Trenches, for startups contains valuable takeaways from the VC AMA Series, and is available for download.

151. Nine startups in the Startup Lounge announced new products based on Google Cloud in the Startup Lounge.

152. We announced the winners of the 2023 Google Cloud Customer Awards in Government and Education Congratulations to: Federal Emergency Management Agency (FEMA), United States Postal Service (USPS), United States Air Force Research Lab (AFRL), State of Hawaii DHS, Salk Institute for Biological Studies, Hawaii Department of Transportation (HDOT), New York State Department of Environmental Conservation (NY DEC) and Minnesota Department of Public Safety (MN DPS).  

Partners

153. We recognized our annual Partners of the Year.

154. A new Google Cloud Generative AI Partner Initiative provides partners with AI journey maps and learning assets, and separate “Services” and “Build” tracks for partners working with gen AI.

155. DocuSign is working with Google Cloud to pilot how Vertex AI could be used to help generate smart contract assistants that can summarize, explain and answer what’s in complex contracts and other documents.

156. SAP is working with us to develop new solutions utilizing Google Cloud foundation models via Vertex AI and SAP data, beginning with solutions to enhance customers’ sustainability initiatives and improve safety and manufacturing processes for the automotive industry. 

157. Workday’s applications for Finance and HR are now live on Google Cloud, and they are working with us to develop new gen AI capabilities within the flow of Workday. 

158. We forged new or expanded partnerships with data providers such as Acxiom, Bloomberg, CoreLogic, Dun & Bradstreet, Equifax, NIQ and TransUnion to provide access to their datasets from BigQuery. 

159. Technology and solutions in the new Data and AI Cloud for Supply Chain helps customers build data-driven supply chains.

160. The new Industry Value Networks (IVN) initiative combines expertise and offerings from systems integrators (SIs), independent software vendors (ISVs) and content partners to create comprehensive, differentiated, repeatable, and high-value solutions that accelerate time-to-value and reduce risk for customers.

Next ‘24!

Last but not least, we’re excited to do it all over again, but even bigger next year so please save the date:

161. Google Cloud Next ‘24 will take place April 9-11 in Las Vegas. 

That was a lot! A huge thank you to our Google Cloud team, customers, and partners who came together to make this all happen. Together, we’re building the new way to cloud.