Cloud

When banks and other financial institutions make their products available via non-banks, we call that Embedded Finance. Think, for example, of a retailer offering financing for people buying goods; or a budgeting app gathering bank transactions to help you categorize and understand your spending patterns. The finance – the money management – has been embedded into the retailer or the budgeting experience.

This last example falls under the Data Exchange category of Embedded Finance , and it’s often also called Open Banking. In many countries this active data exchange is actually mandated by law, with specific Open Banking standards encoded into law for all financial institutions. As of this blog post, there are approximately 37 countries with such laws already in place or currently being worked on.

In an Embedded Finance scheme there are three main players: The financial institution offering their products and services, a third party provider (TPP) making use of those products or services, and an end user who is a customer of the financial institution and is interacting with the TPP.

The most important ingredient for these schemes is consumer trust. If a consumer does not trust that their data will be safely shared with a third party, and that they can easily understand the terms they have agreed to and stop sharing their data whenever they want to, consumers will simply not participate. When a consumer agrees for their data to be shared, this is called consent.

What components do I need for a successful Embedded Finance solution?

Banks’ products and services are made available to other parties via APIs, thus a fundamental component is an API Management platform. An API Management Platform lets you design, secure, analyze, and scale APIs anywhere with visibility and control.

Also, we need a component to manage consents given by end users. Consent Management refers not only to the capability of checking whether a certain action by a third party (eg: access an end user’s account balance) is valid according to the consent granted by the end user to that TPP,  but also the availability of tools for end users to check existing consents, modify them or revoke them, as well as tools for financial institutions to perform similar actions on behalf of their customers.

Finally, we need an Identity Platform capable of authenticating an end user and establishing their identity.

A Google Cloud based embedded finance solution uses Apigee as the API management platform, Identity Platform or your own identity platform of choice, and a consent management service (CMS).  Google Cloud has partnered with two leading providers in the consent management space: Clarip and CloudEntity.

How do these components work together?

The main roles and responsibilities of each of the three components are quite clear, but there is a certain degree of flexibility on how to orchestrate the interactions between the components, and which component performs which function when there is an overlap of functionality between them.

Google Cloud has tested the following interaction patterns:

Interaction Pattern #1: Apigee as the main orchestrator

In this interaction pattern, Apigee interacts with both the consent management service and the identity platform, coordinating end user authentication with the identity platform, and authorization and consent with the consent management service. Apigee acts as the “broker” between users, their identity, and their consent.

Interaction Pattern #2: The consent management service abstracts the identity platform.

In this case, the consent management service manages all the interactions with the identity management platform, and the identity platform acts as the “broker”. Apigee only interacts with the consent service.

The three components share a common pool of client apps and end users. One of the components is typically designated as the source of truth for these entities, while the other components that need to know about these entities synchronize that information from the source of truth. 

The identity platform is typically the source of truth for end user identities and the consent management component will synchronize that information by pulling it from the identity platform. 

The consent management service is the source of truth for user opt-in and current consent status. Apigee will either check with the service the validity of a consent every time a request comes through, or to make it more efficient, it will validate the main attributes of the consent based on information stored in Apigee. To maintain consistency, the consent management service should tell Apigee to invalidate any tokens associated with a consent when it is revoked via other means (for example, if an end user revoked consent from a consent dashboard).

For client apps, either Apigee or the consent management component can act as the source of truth, with the other component synchronizing the information. In the context of Open Banking in particular, standards often mandate that financial institutions support Dynamic Client Registration. This means having endpoints that new clients can use to register themselves as clients for the Open Banking APIs. In this case, the synchronization of clients can easily be incorporated into Dynamic Client Registration.

Let’s look now into more detail about how the three components interact. There are two user journeys that are particularly relevant:

Authentication/Authorization/Consent

This is the beginning of the typical consumer client journey, when an end user starts using a third party app that requires access to that user’s data. The third party app needs to authenticate itself, and request authorization to access the user’s data. This will trigger the authentication of the end user and consent granting: Consent granting being the process of, the user authorizing the operations the app will execute on their behalf, and establishing the list of accounts on which these operations will be performed. The authorization is represented by an access token, which the app will need to include in all subsequent requests.

This is quite a complex journey with many steps involved in order to make this as secure as possible. To make it easier to understand, we will simplify some of the steps (apologies in advance to you OIDC/FAPI experts out there!).

At a high level, these are the steps involved in this user journey:

End user starts using an app

The app requests permission to access user’s data

The identity platform (IdP) authenticates the end user

The consent management service obtains consent from end user

An access token is issued for the app, encapsulating the authorized permissions, accounts, etc. A refresh token is also issued – the access token normally has a short life (measured in minutes), but the app can use the refresh token, which typically lasts much longer (days or months), in order to obtain a new access token without the need of going through the end user authentication/authorization/consent process again.

The way these steps are orchestrated varies depending on the interaction pattern among the components.

Pattern #1: Apigee orchestrates all interactions
When the client requests permission, Apigee will record the request and then redirect the client app to the IdP login page. Once the end user logs in, the IdP will return an authorization code to the callback URL that Apigee registered in the IdP configuration.

Apigee will then redirect the client app to the Consent Management service, so that the consent form is displayed. Once the user grants consent, selects the desired permissions, and selects the accounts it will apply to, the Consent Management Service will update Apigee with the status of the consent, via a callback Apigee registered in the consent configuration.

Now that the end user has authenticated and granted consent, Apigee can now issue an authorization code, and associate all the necessary information, for instance, details about the consent, the authorization code issued by the Identity Platform (IdP), the identification of the end user in the IdP, etc. Apigee will return this authorization code to the client app via a callback endpoint that the app provided when registering with Apigee.

The app will then exchange the authorization code for an access token. Apigee will obtain an access token from the IdP, abd update the status of the consent before issuing an access token to the client.

Pattern #2: The Consent Management Service is the center of the Architecture
In Pattern #2, Apigee will only communicate with the Consent Management Service, which acts as an authorization server for the app and will also coordinate all interaction with the IdP.

When the client requests permission, Apigee will forward the request to the CMS. The CMS will record the request and then redirect the client app to the IdP login page. Once the end user logs in, the IdP will return an authorization code to the callback the CMS registered in the IdP configuration.

The CMS then displays the consent form. Once the user approves the consent and selects the desired permissions and accounts it will apply to, the Consent Management Service will return an authorization code to Apigee, via a callback Apigee registered in the CMS configuration.

From now on, the flow is the same as explained above. Apigee will issue an authorization code, associate all the necessary information and will return this authorization code to the client app via a callback endpoint. The app will then exchange the authorization code for an access token. Apigee will obtain an access token from the CMS, before issuing an access token to the client.

Let’s now look at the other relevant user journey:

Access shared data 

Once the third party app has been authorized, it will then start requesting that end user’s data. For instance, it may request the list of accounts or the balance of a particular account. 

When that request goes through, a series of checks need to be performed: Is this third party app still authorized? Has it been granted the right permission for the operation being requested? Is the account involved included in the list of accounts authorized by the end user? Apigee can perform all these checks by inspecting the access token presented by the third party app. 

There is, however, one other check which cannot be done simply by inspecting the token. Imagine for a moment that in-between the user granting consent and the app requesting the user’s data, the end user decided to revoke that consent. This could be done by other means than the app itself, for instance by checking all consents granted to all apps in a consent dashboard accessible via other channels such as an “Internet Banking” site. When the app requests the user data, the token is still valid, however the consent has already been revoked. Therefore, in this user journey, Apigee will check with the Consent Management service to check if the consent is still valid. This adds a bit of overhead, which can be minimized by other means. For instance, one popular optimization is to make sure that the Consent Management service asks Apigee to revoke any access or refresh tokens associated with a given consent when that consent is revoked via other channels. If that guardrail is in place, then Apigee can determine if a request should be allowed simply by validating the access token and using the consent information stored alongside the token.

The following diagram summarizes the interactions required between Apigee and the Consent Management component. The interaction is essentially the same in both interaction patterns.

What’s next?

In this article we have described how Apigee, a Consent Management service, and an Identity Platform are the fundamental ingredients of a successful and secure Embedded Finance solution. We have also looked at how these components interact with each other to keep user consent consistent and available. Both interaction patterns described are valid and have been proven to work. So which one may be right for you? The answer ultimately depends on your particular requirements, the choice of consent management service, and your choice of identity platform. As a rule of thumb, if you have multiple application experiences, identity systems, and sources of consent, an Apigee-centric architecture is a great solution. If you are looking at a Consent Management service in the context of an Open Banking implementation mandated by a centralized authority (eg: Open Banking Brazil, Consumer Data Standards in Australia), CloudEntity is a good choice for consent management, as their product is also Certified Financial-grade API (FAPI) OpenID Provider, with profiles for the various Open Banking standards. CloudEntity integrates with Apigee using Pattern #2. 

For more details on both Clarip and CloudEntity’s offerings and how their particular solution works with GCP please check:

Clarip: Universal Consent Management Platform – GDPR

CloudEntity: Certified OpenBanking  & Consent (UK, Brazil and Australia) at scale

For more details on Apigee API Management, see https://cloud.google.com/apigee

You can also learn about our solutions for retail banking or contact sales.

Startups are uniquely adept at solving difficult challenges, and Google is committed to partnering with these organizations and delivering technology to help them do so as they start, build, and grow. Over the past year, we’ve deepened our focus on helping startups scale and thrive in the cloud, including launching new resources and mentorship programs, hosting our first-ever Google Cloud Startup Summit, growing our team of startup experts, and more.

With the new year in full swing, I’m excited to roll out several new offerings and updates designed to support startups even more effectively.

First, we will align Google Cloud’s startup program with Google for Startups to ensure startup customers enjoy a consistent experience across all of Google—including Google Cloud infrastructure and services—and to provide founders access to Google mentors, products, programs, and best practices. Going forward, our program will be the Google for Startups Cloud Program.

Next, we’ll deepen our commitment to supporting founders that are just starting out, when access to the right technology and expertise can have a massive impact on their company’s growth trajectory. Early-stage startups are particularly well-positioned to move quickly and solve problems, but they need the ability to scale with minimal costs, to pivot to address a new opportunity, and to leverage expertise and resources as they navigate new markets and investors.  

Supporting early-stage startups is a key goal of the Google for Startups Cloud Program, and today I’m thrilled to announce a new offer for funded startups that will make it easier for these companies to get access to the technology and resources they need. 

Providing new Google Cloud credits for early-stage startups

Starting now, the Google for Startups Cloud Program will cover the first year of Google Cloud usage for investor-backed startups, through series A rounds, up to $100,000. For most startups, this will mean they can begin building on Google Cloud at no cost, ensuring they can focus on innovation, growth, and customer acquisition. In their second year of the program, startups will have 20% of their Google Cloud usage costs covered, up to an additional $100,000 in credits.

This new offering will make it simpler for startups to access to Google Cloud’s capabilities in AI, ML, and analytics, and to rapidly build and scale on Google Cloud infrastructure with services like Firebase and Google Kubernetes Engine (GKE).

Learn more about this new offer and eligibility requirements here.

Connecting startup customers to Google know-how and support

We know that navigating decisions as a fast-scaling startup can be challenging. Last year, we introduced our global Startup Success Team as a dedicated Google Cloud point of contact for startups in our program as they build. Now that this team is fully up and running, we’re expanding it to all qualified, early-stage startups in the Google for Startups Cloud Program. These guides will get to know the unique needs of each startup throughout their two years in the program, and will help connect them with the right Google teams to help resolve any technical, go-to-market, or credit questions along the way. As a customer grows in their usage and expertise with Google Cloud, they’ll be connected to our startup expert account teams to continue their journey.   

The Google for Startups Cloud Program joins Google’s numerous offerings for entrepreneurs. In addition to receiving mentorship, tailored resources, and technical support from Google subject matter experts, participating startups are eligible for additional Google product benefits to help their business including Google Workspace, Google Maps and more. Founders can take advantage of workshops, events, and technical training courses, as well as Google for Startups programsand partner offerings. They can also tap into a supportive network of peers through our new C2C Connect digital community just for founders and CTOs building on Google Cloud. 

Helping startups focus on innovation, not infrastructure

Our goal is to help startups move fast now, without creating technical debt that will slow them down later. With our fully managed, serverless offerings like Cloud Run, Firestore, Firebaseand BigQuery, startups can spend their time on their roadmap, rather than infrastructure management. And as they go from MVP to product to scale, startups don’t need to overhaul their architecture—Google Cloud services scale with them.

That’s how Nylas, a startup focused on business productivity, is able to rapidly scale its platform and support larger, enterprise customers, all while growing its revenue by 5X. FLYR Labs is helping airlines better manage revenue and forecast demand, with a platform powered by Google Cloud data and AI capabilities and running on GKE.Sniip is rapidly growing adoption of its app that helps people more easily track and pay bills, leveraging GKE to scale quickly and Cloud Run to empower their developers.

With Google Cloud, startups benefit from a business and technology partnership to help them build and go to market. We’ll work with founders from the early prototypes to global scale as they expand to new markets. Startups around the world are choosing to build with Google Cloud. Join us and let’s get solving.

In November 2021, we announced the general availability of Tau VMs. Since then, Google Cloud’s Tau VMs with Google Kubernetes Engine (GKE) have unlocked value for many customers who are now using Tau VMs for their production workloads, such as: Ascend, who achieved over 125% higher performance; Nylas, who gained over 40% higher price-performance; and OpenX, who achieved 40% better price-performance while at the same time reducing their application latency by 62%. 

T2D is the first instance type in the Tau VM family and is built on the latest 3rd generation AMD EPYCTM processors, offering 42% higher price-performance compared to general-purpose VMs from any of the leading public cloud vendors. Tau VMs offer a leading combination of performance, price, and full x86 compatibility, offering customers the lowest cost solution for scale-out workloads. Tau VMs are available in predefined shapes, with up to 60vCPUs per VM, 4GB of memory per vCPU, networking up to 32GB and a slew of storage options including Standard, Balanced and Performance PD. Tau VMs are also available as Spot VMs, offering an over 60% discount compared to on-demand pricing. 

For customers looking for advanced container orchestration, GKE delivers high levels of reliability, security, and scalability, and has supported Tau VMs since the day they became available on Google Cloud. Tau VMs are ideal for CPU-bound workloads such as web-serving with encryption, video encoding, compression/decompression, image processing and horizontally-scaled applications. Using Tau VMs along with GKE’s cost-optimization best practices can help lower your total cost of ownership. You can add Tau VMs to new or existing GKE clusters by specifying the Tau T2D machine type in your GKE node-pools through the Cloud console or by using –machine-type in gcloud.

 Here is what some of our customers have to say about Tau VMs:

Ascend provides a unified analytics and data engineering platform, and chose Tau VMs along with GKE to run their data-intensive workload — primarily because of Tau’s absolute performance and price-performance advantage.

“Our core capability at Ascend is bringing together data ingestion, transformation, delivery, orchestration and observability into a single platform. To operate at scale and keep pace with our telemetry data production rates, high single-threaded performance is critical. With Google Cloud’s Tau VMs with Google Kubernetes Engine (GKE), we are able to achieve over 125% higher performance than previous generation families. This has completely changed our ability to query historical metrics. Where previously metric queries against historical data over ranges longer than a couple hours were difficult, we can now easily query data ranges of multiple weeks.” – Joe Stevens, Tech Lead – Infrastructure, Ascend.io

Nylas is a pioneer and leading provider of productivity infrastructure solutions for modern software. In the past year, Nylas has been using GKE in their journey to reinvent their architecture and provide their enterprise customers with a bi-directional universal email sync, security compliance with the highest enterprise standards, and industry-specific machine learning services. 

“For our core application, Google’s Tau VMs with Google Kubernetes Engine delivers over 40% better price-performance than Amazon’s Graviton-based VMs. Further, Tau VMs maintain x86 compatibility and eliminate the need to maintain a separate stack for ARM. We are moving our workload from Amazon Web Services to Google Cloud to take advantage of these benefits.” – David Ting, SVP of Engineering, Nylas

OpenX operates an independent ad exchange. Operating 100% on Google Cloud has enabled OpenX to achieve improved performance, scalability, speed and global reach. 

“At OpenX, our ad-exchange services over 200 billion requests every day. Getting the best combination of performance and price from the infrastructure is critically important for us. We use multiple Google Kubernetes Engine (GKE) clusters across geographic regions with autoscaling to power our ad-delivery components. Running Google Cloud’s Tau VMs with GKE has enabled over 40% better price-performance and 62% latency reduction for our application as compared to the prior generation family. We have made the move to Tau VMs for our application to take advantage of these benefits.” – Paul T.Ryan, CTO, OpenX

We are excited to see Tau VMs adding value for so many of our customers by enabling industry leading price-performance for a variety of workloads. 

If you haven’t tried Tau VMs yet, give them a try today in our Iowa, Netherlands and Singapore regions and move your production workloads to Tau VMs. Tau VMs will be arriving in additional regions and zones in the coming weeks. You can provision GKE node pools based on Tau VMs and explore how you can take advantage of improved price-performance for your scale-out containerized workloads. 

To get started, go to the Google Cloud Console, select Google Kubernetes Engine, and choose Tau T2D for your GKE nodes. To learn more about Tau VMs or other Compute Engine VM options, check out our machine types and our pricing pages.

Cloud Bigtable is a fully managed service that can swiftly scale to meet performance and storage demands with the click of a button. If you’re currently using Bigtable, you might configure your cluster sizes to perform for peak throughput or programmatically scale to match your workload. Bigtable now supports autoscaling for improved manageability, and in one of our experiments autoscaling reduced costs of a common diurnal workload by over 40%.

You only pay for what you need when autoscaling is enabled; Bigtable will automatically add or remove capacity in response to the changing demands of your workloads. Autoscaling enables you to spend more time on your business and less time managing your infrastructure due to the reduced overhead of capacity provisioning management. Autoscaling works on both HDD and SSD clusters, and is available in all Bigtable regions.

We’ll look at when and how to use this feature, go through a performance analysis of autoscaling in action, and finally see how it can impact your database costs.

Enabling Autoscaling

Cloud Bigtable autoscaling is configured at the cluster level and can be enabled using the Cloud Console, the gcloud command-line tool, the Cloud Bigtable Admin API, and Bigtable client libraries.

With autoscaling enabled, Bigtable automatically scales the number of nodes in a cluster in response to changing capacity utilization. The business-critical risks associated with incorrect capacity estimates are significantly lowered: over-provisioning (unnecessary cost) and under-provisioning (missing business opportunities).

Autoscaling can be enabled for existing clusters or configured with new clusters. You’ll need two pieces of information: a target CPU utilization and a range to keep your node count within. No complex calculations, programming, or maintenance are needed. One constraint to be aware of is the maximum node count in your range cannot be more than 10 times the minimum node count. Storage utilization is a factor in autoscaling, but the targets for storage utilization are set by Bigtable and not configurable. 

Below are examples showing how to use the Cloud Console and gcloud to enable autoscaling. These are the fastest ways to get started.

Using Cloud Console

When creating or updating an instance via the Cloud Console you can choose between manual node allocation or autoscaling. When autoscaling is selected, you configure your node range and CPU utilization target.

Using command line

To configure autoscaling via the gcloud command-line tool, modify the autoscaling parameters when creating or updating your cluster as shown below.

Updating an existing cluster:
“`
gcloud bigtable clusters update –instance my-instance my-cluster-c1 –autoscaling-min-nodes 1 –autoscaling-max-nodes 10 –autoscaling-cpu-target 60
“`

Creating a new cluster:
“`
gcloud bigtable clusters create –instance my-instance my-cluster-c1 –zone=asia-east1-c –autoscaling-min-nodes=1 –autoscaling-max-nodes=10 –autoscaling-cpu-target=60
“`

Transparency and trust

On the Bigtable team, we performed numerous experiments to ensure that autoscaling performs well with our customers’ common workloads. It’s important that you have insight into Cloud Bigtable’s autoscaling performance, so you can monitor your clusters and understand why they are scaling. We provide comprehensive monitoring and audit logging to ensure you have a clear understanding of Bigtable’s actions. You’re able to connect Bigtable activity to your billing and performance expectations and fine tune the autoscaling configuration in order to ensure your performance expectations are maintained. Below is the Bigtable cluster monitoring page with graphs for metrics and logs for the cluster.

When is autoscaling right for your workload?

Bigtable is flexible for a variety of use cases with dynamic traffic profiles. Bigtable autoscaling may not always be the right configuration for your business, so here are some guidelines for when autoscaling is ideal.

When to use autoscaling

You’re an existing Bigtable user who wants to optimize costs, while maintaining performance for your cluster. For example: diurnal traffic patterns that you might see with online retail.

You’re a new Bigtable user or have a new workload. Provisioning enough capacity to meet unknown use cases is hard.

Your business is growing, and you’re not sure the extent of future growth.  You want to be prepared to scale for any opportunity.

What autoscaling won’t solve

Certain batch workloads. Autoscaling will react to a sharp increase in traffic (a “step” or batch upload of data). However, Bigtable will still need to rebalance the data and traffic against a rapid increase in nodes, and this may cause a performance impact as Bigtable works to rebalance. 

Autoscaling is likely not the correct solution to resolving hotspotting or ‘hot tablets’ in your Bigtable cluster. In these scenarios it is best to review data access patterns and row key / schema design considerations.

Autoscaling in Action

Cloud Bigtable’s horizontal scalability is a core feature, derived from the separation of compute and storage. Updating the number of nodes for a Bigtable instance is fast whether or not you use autoscaling. When you add nodes to your cluster, Bigtable rebalances your data across the additional nodes, thus improving the overall performance of the cluster. When you scale down your cluster, Bigtable rebalances the load from the removed nodes to the remaining nodes.

With autoscaling enabled, Bigtable monitors the cluster’s utilization target metrics and reacts in real time to scale for the workload as needed. Part of the efficiency of Bigtable’s native autoscaling solution is that it connects directly to the cluster’s tablet servers to monitor metrics, so any necessary autoscaling actions can be done rapidly. Bigtable then adds or removes nodes based on the configured utilization targets. Bigtable’s autoscaling logic scales up quickly to match increased load, but scales down slowly in order to avoid putting too much pressure on the remaining nodes.

Example workload

Let’s look at one of the experiments we ran to ensure that autoscaling performance was optimal in a variety of scenarios. The scenario for our experiment is a typical diurnal traffic pattern: active users during peak times and a significant decrease during off-peak times. We simulated this by creating a Bigtable instance with 30 GB of data per node and performed point reads of 1 kb. 

We’ll get some insights from this experiment using Bigtable’s monitoring graphs. You can access the cluster’s monitoring graphs by clicking on the cluster ID from the Bigtable instance overview page in the Cloud Console.

Having clicked through to the cluster overview page, you can see the cluster’s node and CPU utilization monitoring graphs as seen below. 

The node count graph shows a change from 3 nodes to 27 nodes and back down to 3 nodes over a period of 12 hours. The graph shows the minimum and maximum node counts configured as well as the recommended number of nodes for your current CPU load, so you can easily check that those are aligned. The recommended number of nodes for CPU target (orange line) is closely aligned with the actual number of nodes (blue line) as CPU utilization increases, since scaling up happens quickly to keep up with throughput. As CPU utilization decreases, the actual number of nodes lags behind the recommended number of nodes. This is in line with the Bigtable autoscaling policy to scale down more conservatively to avoid putting too much pressure on the remaining nodes. 

In the CPU utilization graph we see a sawtooth pattern. As it reaches a peak, we can compare both graphs to see the number of nodes is adjusted to maintain the CPU utilization target. As expected, CPU utilization drops when Bigtable adds nodes and steeply increases when nodes are removed. In this example (a typical diurnal traffic pattern), the throughput is always increasing or decreasing. For a different workload, such as one where your throughput changes and then holds at a rate, you would see more consistent CPU utilization. 

On the cluster overview page, we are also able to see the logs and understand when the nodes are changing and why.

To get more insights, you can go to the instance monitoring view. Here we can see even more graphs showing the experiment workload activity. Note the diurnal traffic pattern mentioned above is in line with autoscaling behavior: as throughput increases, node count increases and as throughput decreases, node count decreases.

Cost evaluation

This experiment workload ran for 12 hours. Let’s see how the costs would change for this scenario with and without autoscaling. 

Assume a Bigtable node cost1 of $0.65/hr per node

Comparing the number of nodes and cost when using autoscaling vs scaling for peak: 

15.84 nodes on average for autoscaling / 27 nodes scaled for peak = .587

The number of nodes required is 58.7% of the peak when using autoscaling in this scenario. This is a potential approximate cost saving of 41.3% when using Bigtable native autoscaling in this example.

These savings can be significant when you’re working with large amounts of data and queries per second. 

Summary

Autoscaling with Bigtable provides a managed way to keep your node count and costs aligned with your throughput. 

Get started: Enable autoscaling via the Cloud Console or command line.

Check performance: Keep an eye on your latency with the Bigtable monitoring tools and adjust your node range.

Reduce costs: While maintaining a 60% CPU utilization in our example scenario, the new cost on the diurnal workload was 58.7% of the total when compared to scaling for peak.

1. See Bigtable pricing: https://cloud.google.com/bigtable/pricing
2. ‘Scale for peak’ is the provisioning policy adopted by many DB operational managers to ensure the peak load is supported.

Cloud Bigtable is a fully managed, scalable NoSQL database service for large operational and analytical workloads used by leading businesses across industries, such as The Home Depot, Equifax, and Twitter. Bigtable has more than 10 Exabytes of data under management and processes more than 5 billion requests per second at peak. Today, we’re announcing the general availability of autoscaling for Bigtable that automatically adds or removes capacity in response to the changing demand for your applications. With autoscaling, you only pay for what you need and you can spend more time on your business instead of managing infrastructure. 

In addition to autoscaling, we recently launched new capabilities for Bigtable that reduce cost and management overhead:

2X storage limit that lets you store more data for less, particularly valuable for storage optimized workloads.

Cluster groups provide flexibility for determining how you route your application traffic to ensure a great experience for your customers. 

More granular utilization metrics improve observability, faster troubleshooting and workload management.

 Let’s discuss these capabilities in more detail. 

Optimize costs and improve manageability with autoscaling

The speed of digitization has increased in most aspects of life driving up consumption of digital experiences. The ability to scale up and scale down applications to quickly respond to shifts in customer demand is now more critical for businesses  than ever before.  Autoscaling for Bigtable automatically scales the number of nodes in a cluster up or down according to the changing demands of usage. It significantly lowers your risk of over-provisioning and incurring unnecessary costs, and under-provisioning which can lead to missed business opportunities. Bigtable now natively supports autoscaling with direct access to the Bigtable servers to provide a highly responsive autoscaling solution.

Customers are able to set up an autoscaling configuration for their Bigtable clusters using the Cloud Console, gcloud, the Bigtable admin API, or our client libraries. It works on both HDD and SSD clusters, and is available in all Bigtable regions. 

You can set the minimum and maximum number of nodes for your Bigtable autoscaling configuration in Cloud Console as shown below.

Once you have set up autoscaling, it is helpful to  understand what autoscaling is doing, when and why to reconcile against billing and performance expectations. We have invested significantly in comprehensive monitoring and audit logging to provide developers with granular metrics and pre-built charts that explain how autoscaling makes decisions.

2X the storage limit 

Data is being generated at a tremendous pace and numerous applications need access to that data to deliver superior customer experiences.Many data pipelines supporting these applications require high throughput, and low latency access to vast amounts of data while maintaining the cost of compute resources. In order to meet the needs of storage driven workloads, Bigtable has doubled the storage capacity per node so that you can store more data for less, and don’t have to compromise on your data needs. Bigtable nodes now support 5TB per node (up from 2.5TB) for SSD and 16TB per node (up from 8TB) for HDD. This is especially cost-effective for batch workloads that operate on large amounts of data. 

Manageability at scale with up to eight regions and cluster groups

Businesses today need to serve users across regions and continents and ensure they provide the best experience to every user no matter the location. We recently launched the capability to deploy a Bigtable instance in up to 8 regions so that you can place the data as close to the end user as possible. A greater number of regions helps ensure your applications are performant for a consistent customer experience, where your customers are located. Previously, an instance was limited to four regions.

With a global presence, there are typically multiple applications that require access to the replicated data. Each application needs to ensure that its serving path traffic does not see increased latency or reduced throughput because of a potential ‘noisy neighbor’ when additional workloads need access to the data. To provide improved workload management, we recently launched App Profile Cluster Group routing. Cluster group routing provides finer grained workload isolation management, allowing you to configure where to route your application traffic. This will allow you to allocate Bigtable clusters to handle certain traffic like batch workloads while not directly impacting the clusters being used to serve your customers.

Greater observability

Having detailed insight and understanding of how your Bigtable resources are being utilized to support your business is crucial for troubleshooting and optimizing resource allocation. The recently launched CPU utilization by app profile metric includes method and table dimensions. These additional dimensions provide more granular observability into the Bigtable cluster’s CPU usage and how your Bigtable instance resources are being used. These observability metrics tell you what applications are accessing what tables with what API method, making it much easier to quickly troubleshoot and resolve issues.

Learn more

To get started with Bigtable, create an instanceor try it out with a Bigtable Qwiklab.

Check out Youtube videos for step by step introduction to how Bigtable can be used in real world applications like Personalisation and Fraud detection.

Learn how you can migrate data from HBase to Bigtable

Blockchain technology is yielding tremendous innovation and value creation for consumers and businesses around the world. As the technology becomes more mainstream, companies need scalable, secure, and sustainable infrastructure on which to grow their businesses and support their networks. We believe Google Cloud can play an important role in this evolution.

Building on our existing work with blockchain developers, exchanges, and other companies in this space, we are announcing today a new, dedicated Digital Assets Team within Google Cloud to support our customers’ needs in building, transacting, storing value, and deploying new products on blockchain-based platforms. This new team will enable our customers to accelerate their efforts in this emerging space and help underpin the blockchain ecosystems of tomorrow. 

What We’re Doing Today (and Into the Future)

Blockchain and distributed-ledger-based companies like Hedera, Theta Labs, and Dapper Labs have already chosen to build on top of Google Cloud for scalability, flexibility, and security. Moving forward, Google Cloud’s Digital Assets Team will undertake a number of short- and long-term initiatives to support companies in the digital assets/blockchain ecosystem, including:

Providing dedicated node hosting/remote procedure call (RPC) nodes for developers, allowing users to deploy blockchain validators on Google Cloud via a single click (“click to deploy”).

Participating in node validation and on-chain governance with select partners.

Helping developers and users host their nodes on thecleanest cloud in the industry, supporting their environmental, social, and governance initiatives.

Supporting on-chain governance via participation from Google Cloud executives and senior engineers.

Hosting several public BigQuery datasets on our Marketplace, including full blockchain transaction history for Bitcoin, Ethereum, Bitcoin Cash, Dash, Litecoin, Zcash, Theta, Hedera Hashgraph, Band Protocol, Polygon, XRP, and Dogecoin.

Driving co-development and integration into Google’s robust partner ecosystem, including participating in the Google Cloud Marketplace.

Embracing joint go-to-market initiatives with our ecosystem partners where Google Cloud can be the connective tissue between traditional enterprise and blockchain technologies.

As we build out our team, we’re also exploring opportunities in the future to enable Google Cloud customers to make and receive payments using cryptocurrencies.

Why Partner with Google Cloud

Our customers in this space—both traditional firms seeking to implement blockchain strategies and blockchain-native companies, such as exchanges, app providers, and decentralized platforms—are choosing Google Cloud for three key reasons:

First, their businesses and their developer ecosystems can build on the industry’s cleanest cloud. Growing in a sustainable manner is top-of-mind for many businesses, but is particularly relevant in the blockchain space, where the ability to run and scale sustainably is critical. Google is carbon neutral today, and we’ve announced our goal to run on carbon-free energy, 24/7 at all of our data centers by 2030. We’ve also rolled out the ability for customers to choose a Google Cloud region in which to run based on carbon footprint data.

Second, developers building on blockchain-based platforms can benefit from Google’s world-class developer platform. Google Cloud infrastructure ensures that developers can speed up the delivery of software and data on the blockchain, delivering fast access to applications for users. 

Third, Google Cloud technologies and services will ensure that blockchain-based companies can scale securely and reliably. Google can ensure that data, applications, games, or digital assets like NFTs will be delivered on a stable, secure, and trusted global network.

Google Cloud’s Approach to Blockchain and Digital Assets

Blockchains and digital assets are changing the way the world stores and moves its information —as well as value. As an infrastructure provider, Google Cloud views the evolution of blockchain technology and decentralized networks today as analogous to the rise of open source and the internet 10-15 years ago. Just as open source developments were integral to the early days of the internet, blockchain is yielding innovation and value creation for consumers and businesses. As the technology becomes more mainstream, companies will need scalable, secure infrastructure on which to grow their businesses and support their networks.

As such, we’re applying Google Cloud technology to the blockchain market with the following principles:

Consistent with Google Cloud’s core business: We are specialists in data-powered innovation with leading infrastructure, industry solutions, and other cutting-edge technology. We pursue blockchain projects and partnerships that align with our mission and our expertise.

User trust and governance: Blockchain networks raise novel questions concerning legal compliance and user privacy. We will maintain our commitment to our users through a robust focus on privacy and user trust, as well as an uncompromising focus on compliance with applicable laws.

Network-agnostic: Google’s infrastructure will seek to preserve optionality of networks for the benefit of users. 

We’re inspired by the work already done in the digital assets space by our customers, and we look forward to providing the infrastructure and technologies to support what’s possible with blockchain technologies in the future. If you’re eager to learn more about Google Cloud’s new Digital Assets Team, please reach out to your Google Cloud sales representative or partner manager.

Here at Google Maps Platform, we are embracing low-code development. For more than 15 years, we’ve helped customers like Argos, Dunzo, and Redfin drive success by building custom location-based solutions from scratch. While this approach allows for deep customization, we recognize that not all of our customers have the resources or bandwidth to complete these implementations. One of our most important principles here at Google is “build for everyone.” That’s why we made it easier–regardless of budget or expertise–to build great location-based solutions with Quick Builder. But first, we’ll cover what low-code is and why we’re embracing the movement 

What is low-code development? 

Low code is a visual approach to software development that minimizes the amount of code you need to write from scratch. Low-code tools abstract coding into an intuitive, natural-language based interface that anyone can use, even if you do not know how to code. These tools offer easy access to sophisticated technology, and help enable you to make important decisions about data, design, and functionality. Building demos for stakeholders and demonstrating the tradeoffs of certain decisions is often as simple as clicking a few buttons in the user interface. 

The result is that you get custom, stress-tested and production-ready code faster than if you’d written it yourself. You can copy and paste that output into your codebase verbatim, or use it as a template to customize and extend as much as you’d like. 

Organizations around the world are gravitating to low-code tools because they make designing and developing high-quality software easier than ever. Gartner recently reported that low-code will continue to evolve. And by 2025, half of all new low-code clients will come from buyers outside of IT. With low-code, everyone is enabled to build.hether you know how to code or not, you can explore design decisions easily and quickly.

Introducing Quick Builder

You can now leverage Quick Builder, an intuitive tool to discover, explore, and deploy recommended APIs for your mapping needs. No matter what your technical skill or experience level is, you can use the tool to help grow your business, and deploy to production faster, with less time spent on learning and testing.

We are building out a portfolio of experiences. From within the Google Maps Platform section of the Google Cloud Console, you can find a library of low-code templates to explore. To start, we’ve compiled three common location-based experiences: 

Locator Plus to help drive more people to your location
Locator Plus helps users find your physical locations on a map and allows you to easily include key details, such as hours of operation, available services, user reviews, and photos for each location. Whether you’re a retailer that’s rolling out buy online, pick up in store options, or a bank that wants to help customers find hard to spot ATMs, Locator Plus can easily help your users identify the best location to visit.Address Selection to help speed up checkout and increase conversion
When users have to manually enter their addresses, mistakes can lead to lowered conversions and poor user experiences like undeliverable packages or failed credit card charges. Address Selection, powered by our Place Autocomplete API, can help make your customer sign-up and checkout experience easier and faster, by instantly suggesting accurate, well-formatted  addresses with just a few keystrokes. Neighborhood Discovery to augment your maps with local knowledge
For real estate or travel companies, your users want to get a sense of what’s in the neighborhood: proximity to work, a favorite restaurant, or coffee shop. With Neighborhood Discovery, you can create a hyperlocal experience by displaying a dynamic map of tailored, recommended amenities and attractions.

Quick Builder is a great tool for visual learners. You can add your data, play with a real-time preview, and decide which of the Google-suggested designs best fit your business needs. When you’re happy, you can export custom sample code or use our Cloud Hosting feature to embed the solution in your website with just a single line of code. With Quick Builder, you can get personalized, production-ready code and deploy to a production environment more quickly than ever before.

Looking to the future

This is just the beginning. We’re constantly thinking of ways to help our users get the most out of their Google Maps Platform deployments. Stay tuned for more low-code solution templates to be added to the Quick Builder library. To learn more about Quick Builder, check out our MapsOnAir webinar. 

For more information on Google Maps Platform, visit our website.