Cloud

Many websites and mobile apps use Google Analytics to measure and analyze the behavior of visitors on their platforms. GA4 is the newest version of Google Analytics. It provides a number of new capabilities, such as an export to BigQuery (BQ) that’s available to all GA users, and a way to stream data to BQ in near real time. This post provides an overview of four ways to ingest events for Discovery solutions using GA4 integrations: Google Tag Manager (GTM) GA4, GA BQ Historical ingestion, GA4 BQ Streaming, and Cloud Functions for Firebase.

Events supported by GA4

Cloud Retail can use the following GA4 event types:

add_to_cart (add-to-cart in Retail)

purchase (purchase-complete in Retail)

view_item (detail page view in Retail)

view_item_list (maps to the Cloud Retail search event if combined with a search query )

What about home page views? See more.

GTM GA4 [Availability: GA]

Many people use Google Tag Manager to configure and deploy measurement tags on their websites. Tag Manager provides an easy way to manage and test multiple tags without needing to do many server-side code changes to a site. Cloud Retail has its own first-party tag called ‘Cloud Retail’. If you are sending data via the data_layer to Google Analytics (see the Event reference), the Cloud Retail tag can re-use that same data and send it to the Cloud Retail API.

All you have to do to use the Cloud Retail tag with GA4 is:

Set up a few key parameters (API key, project number)

Choose the ‘Data Layer’ as your desired source for user events

Set up triggers for the Cloud Retail tag – you can re-use your existing GA4 triggers: Set up triggers 

More details: https://cloud.google.com/retail/recommendations-ai/docs/record-events#gtm.

Who is this for: Customers already using GTM, especially those with GA4. Additionally, if you use the Cloud Retail tag you can attach extra dimensions, such as a search query and attribution token, that might not be available if you only use a server-side GA integration (described below).

Who is this not for: Customers not using GTM or not wanting to modify their clients. GTM also does not support mobile events – you would have to ingest these through another integration.

GA4 BQ Historical Import [Availability: Public Preview]

One of the exciting new features introduced with GA4 is the ability to export user events to BigQuery – available to all GA accounts (previously this was restricted to only GA360 users). Unlike the GTM approach above, GA4 BQ will have both mobile and web events and requires no client side changes. To take advantage of this feature, Cloud Retail natively supports importing historical user events via GA4 BQ.

To try this feature out you will need to reach out to your support team to allowlist your project for it. After being allowlisted, the following steps are required:

Enable DataFlow API (this is used to transform the data from GA4 to Cloud Retail).

Grant the following permissions to service accounts used by Cloud Retail:

service-{Project_Number}@google-cloud-sa-retail.iam.gserviceaccount.com hasiam.serviceAccounts.actAs permission on the Compute Engine default service account

If the BQ is in another project than your Retail solution, you will need to grant both the Retail and Compute Engine service account Bigquery Admin role.

Call theRetail API import with data_schema=’ga4′, or go to the Retail console Import tab and choose GA4 as your BQ user events schema.

GA4 BQ exports shards each day into a separate table, so you will need to import each day using a separate import call. 

Caveats: GA4 BQ has limitations on what jobs you can perform at no charge such as the number of events that can be exported, learn more. Data Flow also incurs costs on the end use. See Data Flow pricing.This cost is amortized by subsequent Retail model training and prediction costs. 

Who is this for: Any customer wanting to import their historical events.

Who is this not for: This solution can work for most people, but it only has the ability to import events in bulk. You will need to use GTM or GA4 BQ Streaming to get real-time events. GA4 also does not allow backfilling data into BQ, so in order to have historical data, you have to have the BQ export set up prior to integrating with Cloud Retail. 

GA4 BQ Streaming [Availability: Public Preview]

In addition to supporting a batch export to BQ, GA4 also supports streaming events into BQ in near real-time. Unlike the GTM approach above, GA4 BQ will have both mobile and web events and requires no client-side changes. To take advantage of this capability, we have architected a system on top of Cloud Functions and Cloud Scheduler that periodically polls data from GA4 BQ and streams them to Cloud Retail API. 

The Cloud scheduler will trigger a Cloud Function to run every one minute, and pull the most recent data and send it to Cloud Retail. Detailed instructions are in our customer support Github. 

By default, for home page view events, the import will ingest page views with a url that has no suffix after “/” e.g. myhomepage.com or myhomepage.com/. To ingest home page views that do not match against these, you will need to reach out to your support team.

To try this feature out you will currently need to reach out to your support team to allowlist your project for it. 

Caveat: Given that events might arrive out of order and no timestamp exists for when the data was written, the function might pull duplicate events into Cloud Retail (but our models can handle this). GA4 BQ streaming also provides no SLOs on e2e latency. That being said, the latency from our own empirical analysis suggests latency on the order of less than 1 minute.

Who is this for: Customers who do not use GTM and don’t want to do client-side changes. This is also ideal for customers who have mobile apps, given that GTM won’t cover those.

Who is this not for: Customers who need SLO-backed real-time events and don’t want to deploy a server-side solution – GTM is a better option in this case.

Cloud Functions for Firebase [Availability: Public Preview]

From a functional point of view, there are situations where minimizing the time an event captured by GA4 occurs and enriching the Retail AI models can be key, especially at the customer’s native apps level where, for many retailers, the traffic from the applications represents a high percentage of the global traffic.

To help guide  you through this section, it is helpful to understand how the data will flow through the system:

The native Android or iOS  app generates an analytics event

The Cloud Functions for Firebase trigger for the analytics event is activated.

The Cloud Function configured to handle the event begins processing, and sends data to the Cloud Retail API

Cloud Retail receives the data and processes it for consumption in models and serving systems.

In this section we are going to show an implementation strategy based on the principles of event driven architecture.  The key idea is to extract a GA4  event generated on the mobile app and produce a Recommendations AI event to inject into the Cloud Retail system in real time.  This will be achieved through the following 5 steps:

Step 1: Adding Google Analytics for Firebase SDK

Step 2: Sending a Google Analytics event from your app, and marking it as a conversion.

Step 3: Configuring a Cloud Functions for Firebase trigger to send Analytics events to Google Cloud

Step 4: Using a Google Cloud Function to process the Analytics Events

Step 5: Deploying the Cloud Functions for Firebase

Let’s look at the setup steps in more detail.

Configuring Firebase to send Analytics events to Google Cloud

Before proceeding to this step, we need to log the event via the Google Analytics for Firebase SDK – learn more. 

Once we have instrumented Firebase to log Analytics events, we can configure Firebase to send those events in real-time by going to the Firebase console and marking all relevant events as conversions, as shown in the following figure. Events not marked as a conversion are not sent in real time.

The name of the event is very important because we will need it later for the configuration of the component that should capture it.

At this point, Firebase will generate an event with a JSON payload with the appropriate user and the event dimensions as we will see later. 

Using a Google Cloud Function to process the Analytics Events

Now it’s time to configure the Cloud Function that will process the event. Cloud Functions is a scalable, pay-as-you-go function as a service (FaaS) and it’s the easiest way to process the Analytics events.

Under the hood, the Firebase project that’s linked to your GA4 property is also a Google Cloud project. They share the same project ID, and you can open the same project in both the Firebase console and the Google Cloud Console. The Cloud Function will be deployed there. Cloud Functions can be triggered by different event types. In this case, a Google Analytics for Firebase event is the type of event we need. So, for every conversion event generated by Google Analytics  we will trigger our Cloud Function which will receive and process that event data.

In the case of Java language, a raw Cloud Function implementation could be used as you can see in the following code skeleton:

Now, it’s time to write the code to generate a Cloud Retail event. Before doing so, you have to take into account the different elements you will see in the JSON payload. One of the caveats of the status of the Google Analytics for Firebase trigger is that it’s hard to get the product items information. That information is key in order to generate Cloud Retail events where the product item is mandatory for events like add-to-cart or purchase.

Since we cannot directly access the product items information, it’s necessary to add the product item ad-hoc. One of the recommended workarounds is the Analytics parameters. For the event types that only require one product, the item_id could be used. For the purchase event, one strategy could be to use multiple instances of the custom property purchased_items_i, where i represents an index from 1 to N, providing the product identifiers and also the quantity of the products in the cart. 

Now, we will demonstrate how to use Java to send events to the Cloud Retail API. You will see a full example of in  this Github repo but basically, you have to take into account the following key aspects:

To use a Cloud Retail API, you need to get the right credentials to the model. You need to get a GoogleCredentials using a credential in its JSON representation. You need the public key of the service account with the right permissions (retail editor and token creator) to interact with the model. As you can see in the full example, the Google OAuth Library is needed. 

A UserEvent object has to be created to be sent using the UserEventServiceClient.

The method you need to send a user event to Cloud Retail is the write method. To write a user event using the API, you need to set up an instance of the UserEventServiceClient.

You will see that the builder pattern is omnipresent when it comes to building a UserEvent type object, but here you can see a reference example to build a UserEvent corresponding to the detail-page-view event:

Once we have the UserEvent and the credentials in GoogleCredentials, it’s time to send the event to the Cloud Retail API. Another parameter that is needed is the model URI that follows the pattern projects/<your_recai_project_id>/locations/global/catalogs/default_catalog

Home Page Views are not supported natively in GA4, so you will need to extract them from existing events such as Page Views or Promotion Views. 

Deploying the Cloud Function for Firebase

Now It’s time to deploy your Cloud Function. You have two options: the gcloud CLI and the Google Cloud Console.

CLI

First al all, modify the cloudfunctions.allowedIngressSettings organization policy:

Next deploy your Cloud Function. Below you can see an example showing the key parameters:

To get more information about all the Cloud Function parameters you can visit the gcloud functions reference. 

Parameter

Description

entry-point

Full package name and the main class that represents the Cloud Function entry point.

trigger-resource

Conversion event path from which notifications should be received.

trigger-event

providers/google.firebase.analytics/eventTypes/event.log. This is a fixed value corresponding to the trigger type for Firebase for analytics. 

Cloud Console

In addition to the gcloud CLI tool. You can use the Google Cloud Console to deploy your Cloud Functions as you can see in the following pictures.

Using the Google Cloud Project that corresponds to your Firebase Project, go to organization policies to set the cloudfunctions.allowedIngressSettings policy to ALLOW_INTERNAL_ONLY:

After that, select Cloud Functions in the navigation menu. Click the Create Function button to enter a deployment wizard to deploy your Cloud Function.

First, select the first generation, the Cloud Function name and the region. After that, select the event type (Firebase for Analytics). The log event name must be equal to the Firebase conversion event. After that, it’s time to provide your Cloud Function. In the following picture you can see that you can upload your fat jar in a zip format through a custom bucket that you can create using the console.

After that, you will see your Cloud Function deployed. It’s very useful to have the key metrics, the logs, and the possibility to test your Cloud Function.

Who is this for: Customers who want to enrich Cloud Retail solutions using the events coming from the native apps.

Caveats: The Firebase for analytics trigger event is not able to capture the product item information, so the product items must be injected specifically. Cloud Functions also incur costs on the end user. See Cloud Functions pricing. However, this cost is amortized by subsequent Retail model training and prediction costs.

Future directions: Cloud functions are evolving to their second version, where Eventarc standards will be adopted and the product items information might be provided out of the box.

Cloud Retail Home Page Views

Home Page Views are not supported natively in GA4, so you will need to construct them synthetically.

For GTM, the best way to ingest the Home Page View is to create a separate GTM tag with the data source set as “Variable – Cloud Retail”. Manually construct the Home Page View whenever a visitor lands on your home page, with the trigger being window load, filtered to your home page URL. For example:

In this example, all the required fields are provided by overrides, and the actual variable for the data source is just an empty JSON object.

For GA4, the import will ingest page views with a URL that has no suffix after “/” as Home Page Views. For example, myhomepage.com or myhomepage.com/ would be ingested a Home Page View, but myhomepage.com/home would not. To ingest Home Page Views that do not match against these, you will need to reach out to your support team.

For Firebase, you will need to construct your Home Page Views based on an upstream GA4 event such as page_view or view_promotion. 

As the volume of data that people and businesses produce continues to grow exponentially, it goes without saying that data-driven approaches are critical for tech companies and startups across all industries. But our conversations with customers, as well as numerous industry commentaries, reiterate that managing data and extracting value from it remains difficult, especially with scale. 

Numerous factors underpin the challenges, including access to and storage of data, inconsistent tools, new and evolving data sources and formats, compliance concerns, and security considerations. To help you identify and solve these challenges, we’ve created a new whitepaper, “The future of data will be unified, flexible, and accessible,” which explores many of the most common reasons our customers tell us they’re choosing Google Cloud to get the most out of their data.

For example, you might need to combine data in legacy systems with new technologies. Does this mean moving all your data to the cloud? Should it be in one cloud or distributed across several? How do you extract real value from all of this data without creating more silos?

You might also be limited to analyzing your data in batch instead of processing it in real-time, adding complexity to your architecture and necessitating expensive maintenance to combat latency. Or you might be struggling with unstructured data, with no scalable way to analyze and manage it. Again, the factors are numerous—but many of them accrue to inadequate access to data, often exacerbated by silos, and insufficient ability to process and understand it. 

The modern tech stack should be a streaming stack that scales with your data, provides real-time analytics, incorporates and understands different types of data, and lets you use AI/ML to predictively derive insights and operationalize processes. These requirements mean that to effectively leverage your data assets:

Data should be unified across your entire company, even across suppliers, partners, and platforms., eliminating organizational and technology silos.

Unstructured data should be unlocked and leveraged in your analytics strategy. 

The technology stack should be unified and flexible enough to support use cases ranging from analysis of offline data to real-time streaming and application of ML without maintaining multiple bespoke tech stacks. 

The technology stack should be accessible on-demand, with support for different platforms, programming languages, tools, and open standards compatible with your employees’ existing skill sets. 

With these requirements met, you’ll be equipped to maximize your data, whether that means discerning and adapting to changing customer expectations or understanding and optimizing how your data engineers and data scientists spend their time. In coming weeks, we’ll explore aspects of the whitepaper in additional blog posts—but if you’re ready to dive in now, and to steer your tech company or startup towards success by making your data better work for you, click here to download your copy, free of charge.

As enterprises accelerate their migration to the cloud, they experience more notable mid- and late-phase migration challenges. Specifically, 41% face challenges when optimizing apps in the cloud post-migration, and 38% struggle with performance issues on workloads migrated to the cloud. Further, organizations have also increased reliance on outside consultants and other service providers for early-stage cloud migration tasks to ongoing management post-implementation.1

To help customers through these challenges with a simple, quick path to a successful cloud migration, Google Cloud created our comprehensive Rapid Assessment & Migration Program (RAMP). And we’ve got some exciting developments to share with our customers and partners: 

Expanded focus on post-migration TCO/ROI

Given the complex nature of cloud migrations, we are committed to meeting our customers where they’re at in their cloud journeys and partnering with them to achieve their business goals — be it building customer value through innovation, driving cost efficiencies, or increasing competitive differentiation and productivity. RAMP is a holistic framework based on tangible customer TCO and ROI analyses, that supports our customers’ journeys all the way through: from assessing their digital landscapes across multiple sources including on-prem and other clouds, and identifying prioritized target workloads to building a comprehensive migration and modernization plan. 

Accelerate positive outcomes with expert partners

Customers can also now expect a more streamlined migration experience through our ecosystem of partners who have completed their cloud migration specialization. Last week, we announced industry-leading updates to our partner funding programs with new assessment and consumption packages that simplify and accelerate our customers’ journey to Google Cloud, at little-to-no cost. These packages offer prescriptive pathways for infrastructure and application modernization initiatives, empowering our partners to support our customers at every stage — from discovery and planning to migration and modernization. 

Through our partner ecosystem, our customers can expect:

Distinct funding packages for assessment, planning, and migration

Faster approval processes for accelerated deployments

More partners eligible to participate in RAMP and access these new funding packages

Sustainability through migration

Another major focus area for RAMP is helping enterprises optimize their migration planning and maximize their ROI by including their business and technical considerations early in the process and including any sustainability goals they may have. To aid with their sustainability efforts, we are excited to share that customers can now receive a Digital Sustainability Report along with their IT assessments – enabling sustainability to be built into their migration strategies. The report provides actionable insights to measure and reduce their environmental impact, and is based on some of Google Cloud’s own best practices, having been carbon-neutral for decades and looking to run on carbon-free energy by 2030. 

We are committed to solving complex problems for our customers and partners, and these updates are a reflection of the feedback we receive. Simplify your cloud migration strategy today by requesting your free assessment, finding a partner to work with, or talking to your existing partner to get started.

1. Forrester Consulting, State Of Public Cloud Migration; A study commissioned by Google, 2022

Customers who work with data warehouses, running BI on large datasets used to have to pick low latency but trading off freshness of data. With BigQuery BI Engine, they can accelerate their dashboards and reports that connect to BigQuery without having to sacrifice freshness of the data. Using the latest insights helps them make better decisions for the business. BI Engine enables customers to be able to get “formula one” performance for their queries across all BI tools that connect with BigQuery, thereby helping them leverage existing investments. 

Last year, we launched a preview of BigQuery BI Engine, a fast in-memory analysis service that accelerates and provides sub-second query performance for dashboards and reports that connect to BigQuery. BI Engine works with any BI or custom dashboarding tool. This was designed to help analysts identify trends faster, reduce risk, match the pace of customer demand, and improve operational efficiency in an ever-changing business climate. With this launch, customers were able to build fast, interactive dashboards using any of the popular tools like Looker, Tableau, Sheets, PowerBI, Qlik or even any custom application.

And our customers have realized this value quickly. “We have seen significant performance improvements within BigQuery after implementing BI Engine. Our views and materialized views have been especially improved after implementing BI Engine.” says Yell McGuyer, Data Architect at Keller Williams Realty.

Today, we are very excited to announce the general availability of BigQuery BI Engine for all BI and custom applications that work with BigQuery! 

BI Engine Acceleration works seamlessly with BigQuery 

Native Integration with the BigQuery API. BI Engine natively integrates with the BigQuery API, which means that if your dashboards use standard interfaces like SQL, BigQuery APIs or JDBC/ODBC drivers to connect to BigQuery, then BI Engine is automatically supported. No changes are required for applications or dashboards to get sub-second, scalable dashboards up and running. If you run a query with BigQuery and if it can be accelerated, it will be accelerated with BI Engine.

Intelligent Scaling. Customers do not have to worry about efficiently using the memory reservation, BI Engine does it for you based on the access patterns. BI Engine leverages advanced techniques like vectorized processing, advanced data encodings, and adaptive caching to maximize performance while optimizing memory usage. It can also intelligently create replicas of the same data to enable concurrent access.

Simple Configuration. The only configuration needed when using BI Engine is to set up memory reservation, which is provided in a fine-grained increment of 1GB each. 

Full Visibility. Monitoring and logging are critical for running applications in the cloud and to gain insight into performance and opportunities for optimization. BI Engine integrates with familiar tools such as Information Schema for job details (e.g. aggregate refresh time, cache hit ratios, query latency etc) and Stackdriver for monitoring of usage. 

Getting started with BI Engine

BI Engine is now available in all regions where BigQuery is available. 

You can sign up for a BigQuery sandbox here and enable BI Engine for your project. Feel free to read through the documentation and quick-starts guides for popular BI tools. You can also watch the demo from Data Cloud Summit to see how BI Engine works with BI tools like Looker, Data Studio and Tableau. 

If you’re a partner with an integration to BigQuery, consider joining the Google Cloud Ready – BigQuery initiative. You can find more details about the program here.