Cloud

Editor’s note: The post is part of a series showcasing tech companies and data providers that are Built with BigQuery.

In recent years, changes in consumer privacy regulations have created disruption within the media and advertising ecosystem. Publishers have felt some of the biggest impact, needing to change the way they think about ad monetization so they can continue to respect user privacy, reduce reliance on third parties, and build sustainable revenue growth. 

Key to helping publishers navigate these challenges is their data management and collaboration technologies, which enable them to utilize private identity data, extract meaningful business insights, and safely scale data activation. 

Optable is the maker of an end-to-end data clean room platform for the advertising industry that integrates with BigQuery and enables audience activation and insights through connections with downstream systems.

Optimizing data management for monetization 

The methods and systems that support audience-focused ad planning, activation, and measurement have undergone a massive shift. As a result, publishers are needing to reevaluate their data strategies, shifting their investments to cloud computing and big data. Unfortunately, a lack of innovation in data management platforms (DMPs) for publishers has stifled revenue growth and created inefficiencies in operations. This lack of ROI in legacy DMPs comes from the following challenges:

Platforms can’t deliver insights across known users, anonymous traffic, and ad data. A shift across the media landscape to subscription-based approaches and user consent has meant publishers have an influx of first-party data from their audiences. This change means they need new types of privacy-centric tools to not only manage this data, but also combine it with other datasets to derive insights for both internal business operations as well as advertising partners.
Identity management can consume the limited resources of modern publishers and media companies. As media consumption has continued to fragment, identifying your audience to deliver relevant media and advertising has become a struggle. Many customer data platforms (CDPs) and legacy DMPs rely on systems that are too rigid and profile-based. This makes it difficult to understand true audience reach and attributes, and leaves analytics teams with incomplete or error-prone data sets.
There’s a lack of privacy-centric data monetization tools and integrations for publishers. Given the rapid onset of privacy changes, many publishers are navigating a reduction in addressability, signal loss, and programmatic revenues. Current technologies lack interoperability with second-party datasets, advertising ID providers, and ad delivery systems to support direct sales growth.
Legacy solutions are not extendable and interoperable across a growing cloud ecosystem. Data management solutions have not kept up with innovation in the cloud computing ecosystem. This makes data extensibility into native cloud tools for things like modeling, custom queries and data visualization is difficult without cumbersome data copying. Additionally, most data management platforms lack native functionality with cloud-based clean room capabilities, such as data analysis rules and egress restrictions, across all major cloud providers. 

Privacy-safe addressable ad products for publishers

Optable is a next-generation data management and collaboration platform that is privacy-focused by design and built to harness the power of today’s big data and cloud computing ecosystem. To achieve this, Optable focuses on three key areas of innovation.

1. Composable identity 
The Optable platform simplifies the complexities of processing and analyzing audience identity data in the age of privacy. By creating a flexible identity system, publishers can shape their audience graph for maximum accuracy and addressability. Optable also makes it easy to connect alternative identifiers such as UID 2.0 or ID5 to your audience data so that you can understand the full spectrum of advertiser demand and maximize revenue. It’s also straightforward to enrich audience data through connecting to second-party data sources, such as True Data.

2. Interoperability focused on monetization
Optable makes monetization easy. It’s built the tools and integrations needed to keep up with programmatic ad-tech ecosystem changes. Audience data is fully interoperable through the Google Cloud ecosystem (and beyond), which means it’s easier to activate through ad servers like Google Ad Manager (GAM) or Google DV 360, or use pre-built clean room applications within BigQuery. This combination means that direct sales and ad operations teams can quickly and easily move from ad partner planning, to audience building and insights, to actual campaign activation.

3. Audience insights across all known users and event traffic 
Audience building and analysis with Optable provides a comprehensive 360-degree view without the massive upfront engineering that’s usually required. By allowing audiences to be created, synthesized, and managed using both known user data as well as anonymous events (such as page visits or ad serving events from GAM), operational teams can glean meaningful insights, service unique requests from ad partners, and optimize campaign performance. For example, in the image below, Optable Insights shows customers key information about the size and makeup of their private identity graph as well as the makeup of Traits across their audience.

One of Optable’s early customers, a major North American news publisher, developed a digital-first publication strategy in 2017. As part of this they built an ad-sales strategy focused on privacy-safe first-party data monetization. A key pillar to their strategy is data collaboration through Optable. Through this investment they achieved a 9% annual increase in ad revenue in 2022.

How Optable creates growth and efficiency

Optable on BigQuery empowers publishers to monetize their data assets, even if they lack engineering resources. Optable provides the right tools and integrations so teams can use their data to create ad products. For example, audience data can be exported directly into Google Ads Data Manager for easy activation.

The data clean room is one of the most important technologies in the media and advertising space. The Optable platform gives publishers pre-built applications that are powered by BigQuery data clean room primitives and APIs. These can also easily be extended to power secure data collaboration across other cloud ecosystems using Optable’s ‘Flash Nodes’ and ‘Flash Connectors.’ Flash Nodes allow companies to invite partners to easily onboard their data into a limited version of Optable simply for the purpose of collaborating with that partner; this reduces the friction of setting up a whole new collaboration platform. Likewise, ‘Flash Connectors’ give companies a set of primitives that can be shared with partners who use AWS, Snowflake, and BigQuery so that Optable users can collaborate directly with partners who house their data in those environments without moving any data.

Rather than just focusing on moving data in and out, Optable’s platform is built for a data-warehouse centric world, so publishers can extend their capabilities. Identity graphs can be shaped and custom modeling easily deployed directly in BigQuery. Additionally, the capabilities of the Google Data Cloud enable custom use cases such as data visualization in Looker. To help navigate data security compliance, Optable has built ‘Bring Your Own Account’ functionality to allow customers to utilize a Google BigQuery instance that’s fully controllable.

Optable partners with Google across many areas, including a new integration with the Google Privacy Sandbox APIs that enable publishers to easily onboard and activate audiences as well as enabling marketers to run campaigns via Privacy Sandbox. An early access program enables these capabilities directly through the Optable platform. You can learn more here.

Later this year, Optable is announcing enhancements to its audience data management and collaboration capabilities, including support for ad serving events through Google Ad Manager and the release of the Google BigQuery Connector to allow for zero-copy partner collaboration. For more information on these and other capabilities please visit the Optable website. 

Optable also recently became a Google Cloud Marketplace vendor, so Google Cloud customers can now purchase and implement Optable’s platform directly. Following the recently announced general availability of BigQuery data clean rooms, the Optable and Google teams are working on an integration to unlock more planning, activation and measurement use cases for the media and advertising ecosystem.

The Built with BigQuery advantage

Built with BigQuery helps companies like Optable build innovative applications with Google Data Cloud. Participating companies can: 

Accelerate product design and architecture through access to designated experts who can provide insight into key use cases, architectural patterns, and best practices. 

Amplify success with joint marketing programs to drive awareness, generate demand, and increase adoption. 

BigQuery gives ISVs the advantage of a powerful, highly scalable unified AI lakehouse that’s integrated with Google Cloud’s open, secure, sustainable platform. Click here to learn more about Built with BigQuery

Editor’s note: Today’s blog post was written in collaboration with Google Cloud Premier Partner Virtusa, which specializes in building cloud-native, microservices-based, pre-built solutions and APIs on Kubernetes, as well as machine learning and data-oriented applications, as well as offering managed cloud services and cloud operations design.

In the ever-evolving world of data engineering and analytics, traditional centralized data architectures are facing limitations in scalability, agility, and governance. To address these challenges, a new paradigm, called “data mesh,” has emerged, which allows organizations to take a decentralized approach to data architecture. This blog post explores data mesh as a concept and delineates the ways that Dataplex, a data fabric capability within the BigQuery suite, can be used to realize the benefits of this decentralized data architecture.

Data mesh is an architectural framework that promotes the idea of treating data as a product and decentralizes data ownership and infrastructure. It enables teams across an organization to be responsible for their own data domains, allowing for greater autonomy, scalability, and data democratization. Instead of relying on a centralized data team, individual teams or data products take ownership of their data, including its quality, schema, and governance. This distributed responsibility model leads to improved data discovery, easier data integration, and faster insights.

The illustration in Figure 1 is an overview of data mesh’s fundamental building blocks.

Let’s discuss the core principles of data mesh architecture, then understand how they change the way we manage and leverage data.

Domain-oriented ownership: Data mesh emphasizes decentralizing data ownership and the allocation of responsibility to individual domains or business units within an organization. Each domain takes responsibility for managing its own data, including data quality, access controls, and governance. By doing so, domain experts are empowered, fostering a sense of ownership and accountability. This principle aligns data management with the specific needs and knowledge of each domain, ensuring better data quality and decision-making.

Self-serve data infrastructure: In a data mesh architecture, data infrastructure is treated as a product that provides self-serve capabilities to domain teams. Instead of relying on a centralized data team or platform, domain teams have the autonomy to choose and manage their own data storage, processing, and analysis tools. This approach allows teams to tailor their data infrastructure to their specific requirements, accelerating their workflows and reducing dependencies on centralized resources.

Federated computational governance: Data governance in a data mesh is not dictated by a central authority; rather, it follows a federated model. Each domain team collaboratively defines and enforces data governance practices that align with their specific domain requirements. This approach ensures that governance decisions are made by those closest to the data, and it allows for flexibility in adapting to domain-specific needs. Federated computational governance promotes trust, accountability, and agility in managing data assets.

Data as a product: Data in a data mesh is treated as a product, and data platforms are built and managed with a product mindset. This means focusing on providing value to the end users (domain teams) and continuously iterating and improving the data infrastructure based on feedback. When teams adopt a product thinking approach, data platforms become user-friendly, reliable, and scalable. They evolve in response to changing requirements and deliver tangible value to the organization.

Understanding Dataplex

Dataplex is a cloud-native, intelligent data fabric platform designed to simplify and streamline the management, integration, and analysis of large and complex data sets. It offers a unified approach to data governance, data discovery, and data lineage, enabling organizations to gain more value from their data.

Key features and benefits of Dataplex include data integration from various sources into a unified data fabric; robust data governance capabilities that help ensure security and compliance; intelligent data discovery tools for enhanced data visibility and accessibility; scalability and flexibility to handle large volumes of data in real-time; multi-cloud support for leveraging data across different cloud providers; and efficient metadata management for improved data organization and accessibility.

Steps to implement data mesh using Dataplex

Step 1: Create a data lake and define the data domain.

In this step, we set up a data lake on Google Cloud and establish the data domain, which refers to the scope and boundaries of the data that will be stored and managed in the data lake. A data lake is a centralized repository that allows you to store structured, semi-structured, and unstructured data in its native format, making it a flexible and scalable solution for big data storage and analytics.

The following diagram illustrates domains as Dataplex lakes, each owned by distinct data producers. Within their respective domains, data producers maintain control over creation, curation, and access. Conversely, data consumers have the ability to request access to these lakes (domains) or specific zones (subdomains) to conduct their analysis.

Step 2: Create zones in your data lake and define the data zones.

In this step, we divide the data lake into zones. Each zone serves a specific purpose and has well-defined characteristics. Zones help organize data based on factors like data type, access requirements, and processing needs. Creating data zones provides better data governance, security, and efficiency within the data lake environment. 

Common data zones include the following:

Raw zone: This zone is dedicated to ingesting and storing raw, unprocessed data. It is the landing zone for new data as it enters the data lake. Data in this zone is typically kept in its native format, making it ideal for data archival and data lineage purposes.

Curated zone: The curated zone is where data is prepared and cleansed before it moves to other zones. This zone may involve data transformation, normalization, or deduplication to ensure data quality.

Transformed zone: The transformed zone holds high-quality, transformed, and structured data that is ready for consumption by data analysts and other users. Data in this zone is organized and optimized for analytical purposes.

Step 3: Add assets to the data lake zones.

 In this step, we focus on adding assets to the different data lake zones. Assets refer to the data files, data sets, or resources that are ingested into the data lake and stored within their respective zones. By adding assets to the zones, you populate the data lake with valuable data that can be utilized for analysis, reporting, and other data-driven processes.

Step 4: Secure your data lake. 

In this step, we implement robust security measures to safeguard your data lake and the sensitive data it holds. A secure data lake is crucial for protecting sensitive information, helping to ensure compliance with data regulations, and maintaining the trust of your users and stakeholders.

The security model in Dataplex enables you to control access for performing the following tasks:

Managing a data lake, which involves tasks such as creating and associating assets, defining zones, and setting up additional data lakes

 Retrieving data linked to a data lake via the mapped asset (e.g., BigQuery data sets and storage buckets)

 Retrieving metadata associated with the data linked to a data lake

The administrator of a data lake manages access to Dataplex resources (including the lake, zones, and assets) by assigning the necessary basic and predefined roles.  Metadata roles possess the capability to access and examine metadata, including table schemas. With data roles granted, it gives the privilege to read or write data in the underlying resources referenced by the assets within the data lake.

Advantages of building a data mesh

Improved data ownership and accountability: One of the primary advantages of a data mesh is the shift in data ownership and accountability to individual domain teams. By decentralizing data governance, each team becomes responsible for the quality, integrity, and security of their data products. 

Agility and flexibility: Data meshes empower domain teams to be autonomous in their decision-making, allowing them to respond swiftly to evolving business needs. This agility enables faster time to market for new data products and iterative improvements to existing ones. 

Scalability and reduced bottlenecks: A data mesh eliminates scalability bottlenecks by distributing data processing and analysis across domain teams. Each team can independently scale its data infrastructure based on its specific needs, ensuring efficient handling of increasing data volumes.

Enhanced data discoverability and accessibility: Data meshes emphasize metadata management, enabling better data discoverability and accessibility. With comprehensive metadata, teams can easily locate and understand available data assets. 

Empowerment and collaboration: By distributing data knowledge and decision-making authority, domain experts are empowered to make data-driven decisions aligned with their business objectives. 

Scalable data infrastructure: With the rise of cloud technologies, data meshes can take advantage of scalable cloud-native infrastructure. Leveraging cloud services, such as serverless computing and elastic storage, enables organizations to scale their data infrastructure on-demand, ensuring optimal performance and cost-efficiency.

Comprehensive and robust data governance: Dataplex offers an extensive solution for data governance, ensuring security, compliance, and transparency throughout the data lifecycle. With fine-grained access controls, encryption, and policy-driven data management, Dataplex enhances data security and facilitates adherence to regulatory requirements. The platform provides visibility into the entire data lifecycle through lineage tracking, promoting transparency and accountability. Organizations can enforce standardized governance policies, ensuring consistency and reliability across their data landscape. Dataplex’s tools for data quality monitoring and centralized data catalog governance further contribute to effective data governance practices.

Learn more

By embracing the principles of decentralization, data ownership, and autonomy, businesses can unlock a range of benefits, including improved data quality, greater accountability, and enhanced agility, scalability, and decision-making. Embracing this innovative approach can position organizations at the forefront of the data revolution, driving growth, innovation, and a competitive advantage. Learn more about Google Cloud’s open generative AI partner ecosystem. To get started with Google Cloud and Virtusa and to learn more about building a data mesh using Dataplex, contact us today.

Creating event-driven architectures using Eventarc together with Firestore is an increasingly popular pattern. Recently, the Firestore integration with Eventarc became generally available, adding new functionality. You can now register multiple Cloud Functions in different regions against a multi-regional Firestore database for increased reliability, and there are new event types, including the Auth Context extension for CloudEvents. 

Determining who or what — the user, a service account, the system, or a third-party — is making a modification to a Firestore document as a change event has long been a top-requested feature. With the new Firestore event types with Auth Context extension, events now embed metadata about the principal that triggered a document change in the open and portable CloudEvents format.

Example walkthrough

Let’s say that you want to have different logic to process events in the destinations for different auth contexts (i.e. unauthenticated or system). To set up your trigger, navigate to the Eventarc section of the Google Cloud console. You’ll need to create a new trigger for Firestore using the associated event types that include authentication information. These event types end with the suffix *.withAuthContext. We’ll want to capture newly written entities, so we’ll select  google.cloud.firestore.document.v1.written.withAuthContext events:

You can specify additional filters, which ensures only desirable events from a specified database and collection are delivered. In this case, we filter for events from the (default) database, and for documents of the collection Ops.

On the same screen, you’ll also need to specify a destination. Triggering events can be delivered to any number of supported Eventarc destinations, like Cloud Run, Cloud Functions (2nd gen), and Google Kubernetes Engine. Let’s say we have a Cloud Run service named demo that exposes an HTTP endpoint to receive the events. You can configure your trigger as follows:

That’s it! When any write operation is applied to your (default) database with the collection Ops, a CloudEvent with the Auth Context is delivered to the configured Cloud Run service demo almost immediately. You can inspect the authtype attribute as defined in Auth Context extension to identify unauthenticated and system types as shown in https://cloud.google.com/firestore/docs/extend-with-functions-2nd-gen#event_attributes . 

Next steps

For more information on how to set up and configure Firestore triggers, check out our documentation.

Thanks to both Minh Nguyen, Senior Product Manager Lead for Firestore and Juan Lara, Senior Technical Writer for Firestore, for their contributions to this blog post.

Editor’s note: Today we hear from Atommap, a computational drug discovery company that has built an elastic supercomputing cluster on the Google Cloud to empower large-scale, computation-driven drug discovery. Read on to learn more.

Bringing a new medicine to patients typically happens in four stages: (1) target identification that selects the protein target associated with the disease, (2) molecular discovery that finds the new molecule modulating the function of the target, (3) clinical trial that tests the candidate drug molecule’s safety and efficacy in patients, and (4) commercialization that distributes the drug to patients in needs. The molecular discovery stage, in which novel drug molecules are invented, involves solving two problems: first, we need to establish an effective mechanism to modulate the target function that maximizes the therapeutic efficacy and minimizes the adverse effect; second, we need to design, select, and make the right drug molecule that faithfully implements the mechanism, is bioavailable, and has acceptable toxicity.

What makes molecular discovery hard?

A protein is in constant thermal motion, which changes its shape (conformation) and binding partners (other biomolecules), thus affecting its functions. Structural detail of a protein’s conformational dynamics time and again suggests novel mechanisms of functional modulation. But such information often eludes experimental determination, despite tremendous progress in experimental techniques in recent years.

The chemical “universe” of all possible distinct small molecules — estimated to number 1060 (Reymond et al. 2010) — is vast. Chemists have made probably ten billion so far, so we still have about 1060 to go.

There lie two major challenges of molecular discovery and its endless opportunity: chances are that we have not considered all the mechanisms of action or found the best molecules, thus we can always invent a better drug. 

Atommap’s computation-driven approach to molecular discovery

Harnessing the power of high-performance computing, Atommap’s molecular engineering platform enables the discovery of novel drug molecules against previously intractable targets through new mechanisms, making the process faster, cheaper, and more likely to succeed. In past projects, Atommap’s platform has dramatically reduced both the time (by more than half) and cost (by 80%) of molecular discovery. For example, it played a pivotal role in advancing a molecule against a challenging therapeutic target to the clinical trial in 17 months (NCT04609579) and it substantially accelerated the discovery of novel molecules that degrade high-valued oncological targets (Mostofian et al. 2023).

Atommap achieves this by:

Advanced molecular dynamics (MD) simulations that unveil complex conformational dynamics of the protein target and its interactions with the drug molecules and other biomolecules. They establish the dynamics-function relationship for the target protein, which is instrumental to choosing the best mechanism of action for the drug molecules. 

Generative models that enumerate novel molecules. Beginning with a three-dimensional blueprint of a drug molecule’s interaction with its target, our models computationally generate thousands to hundreds of thousands of new virtual molecules, which are designed to form the desired interactions and to satisfy both synthetic feasibility and favorable drug-like properties. 

Physics-based, ML-enhanced predictive models that accurately predict molecular potencies and other properties. Every molecular design is evaluated computationally for its target-binding affinity, its effects on the target, and its drug-likeness. This allows us to explore many times more molecules than can be synthesized and tested in the wet lab, and to perform multiple rounds of designs while waiting for often-lengthy experimental evaluation, leading to compressed timelines and increased probability of success.

Computation as a Service and Molecular Discovery as a Service

To truly, broadly impact drug discovery, Atommap needs to augment its deep expertise in molecular discovery by partnering with external expertise in the other stages — target identification, clinical trials, and commercialization. We form partnerships in two ways: Computation as a Service (CaaS) and Molecular Discovery as a Service (MDaaS, pronounced Midas), which make it easy and economically attractive for every drug discovery organization to access our computation-driven molecular engineering platform. 

Instead of selling software subscriptions, Atommap’s pay-as-you-go CaaS model lets any discovery project first try our computational tools at a small and affordable scale, without committing too much budget. Not every project is amenable to computational solutions, but most are. This approach allows every drug discovery project to introduce the appropriate computations cheaply and quickly, with demonstrable impact, and then deploy them at scale to amplify their benefits.

For drug hunters who would like to convert their biological and clinical hypotheses into drug candidates, our MDaaS partnership allows them to quickly identify potent molecules with novel intellectual property for clinical trials. Atommap executes the molecular discovery project from the first molecule (initial hits) to the last molecule (development candidates), freeing our partners to focus on biological and clinical validation. 

The need for elastic computing

For Atommap, the number of partnership projects and the scale of computation in each project fluctuate over time. In building structural models to enable structure-based drug design, we run hundreds of long-timescale MD simulations on high-performance GPUs to explore the conformational ensembles of proteins and complexes between proteins and small molecules, each of which can last hours to days. Our NetBFE platform for predicting the binding affinities invokes thousands, sometimes tens of thousands, of MD simulations, although each one is relatively short and completes in a few hours. Atommap’s machine learning (ML) models take days to weeks to train on high-memory GPUs, but once trained and deployed in a project, run in seconds to minutes. Balancing the different computational loads associated with different applications poses a challenge to the computing infrastructure. 

To meet this elastic demand, we chose to supplement our internal computer clusters with Google Cloud.

How to build an elastic supercomputer on Google Cloud

It took us several steps to move our computing platform from our internal cluster to a hybrid environment that includes Google Cloud.

Slurm 

Many workflows in our platform depended on Slurm for managing the computing jobs. To migrate to Google Cloud, we built a cloud-based Slurm cluster using Cloud HPC Toolkit, an open-source utility developed by Google. Cloud HPC Toolkit is a command line tool that makes it easy to stand up connected and secure cloud HPC systems. With this Slurm cluster up and running in minutes, we quickly put it to use with our Slurm-native tooling to set up computing jobs for our discovery projects.

Cloud HPC Toolkit naturally fits our DevOps function into best practices. We defined our compute clusters as “blueprints” within YAML files that allow us to simply and transparently configure specific details of individual Google Cloud products. The Toolkit transpiles blueprints into input scripts that are executed with Hashicorp’s Terraform, an industry standard tool for defining “infrastructure-as-code” such that it can be committed, reviewed, and version-controlled. Within the blueprint we also defined our compute machine image through a startup script that’s compatible with Hashicorp’s Packer. This allowed us to easily “bake in” the software our jobs typically need, such as conda, Docker, and Docker container images that provide dependencies such as AMBER, OpenMM, and PyTorch.

The deployed Slurm cloud system is as accessible and user-friendly as any Slurm system we have used before. The compute nodes are not deployed until requested and are spun down when finished, thus we only pay for what we use; the only persistent nodes are the head and controller nodes that we log into and deploy from.

Batch

Compared to Slurm, the cloud-native Google Batch gives us even greater flexibility in accessing the computing resources. Batch is a managed cloud job-scheduling service, meaning it can be used to schedule cloud resources for long-running scientific computing jobs. Virtual machines that Batch spins up can easily mount either NFS stores or Google Cloud Storage buckets, the latter of which are particularly suitable for holding our multi-gigabyte MD trajectories and thus useful as output directories for our long-running simulations.

Running our workflows on Google Cloud through the Batch involves two steps: 1) copying the input files to Google Cloud storage, 2) submitting the batch job.

SURF

A common pattern has emerged in most of our computational workflows. First, each job has a set of complex input files, including the sequence and structures of the target protein, a list of small molecules and their valence and three-dimensional structures, and the simulation and model parameters. Second, most computing jobs take hours to days to finish even on the highest-performance machines. Third, the computing jobs produce output datasets of substantial volume and subject to a variety of analyses.

Accordingly, we have recently developed a new computing infrastructure, SURF (submit, upload, run, fetch), which seamlessly integrates our internal cluster and Google Cloud through one simple interface and automatically brings the data to where it is needed by computation or the computation to where the data resides.

SURF submits jobs to Google Cloud Batch using Google’s Python API.

We now have an elastic supercomputer on the cloud that gives us massive computing power when we need it. It empowers us to explore the vast chemical space at an unprecedented scale and to invent molecules that better human health and life.

Andrew Sabol supported us from the very beginning of Atommap, even before we knew whether we could afford the computing bills. Without the guidance and technical support of Vincent Beltrani, Mike Sabol, and other Google colleagues, we could not have rebuilt our computing platform on Google Cloud in such a short time. Our discovery partners put their trust in our young company and our burgeoning platform; their collaborations helped us validate our platform in real discovery projects and substantially improve its throughput, robustness, and predictive accuracy.

On a mission to disrupt and transform the fintech industry, Airwallex is constantly innovating, embracing new technologies that can help businesses overcome international money transfer challenges so they can grow without borders. We aim to provide end-to-end solutions and currently offer services related to global payments, treasury and expense management, and embedded finance services. With our proprietary infrastructure, we can remove friction from global payments and financial operations, so that businesses of all sizes can unlock new opportunities for growth.

Our developers are at the heart of everything we do at Airwallex. We follow a “develop first, and deploy fast” mindset, which we believe has benefited our customers, and we are highly focused on finding a good balance between velocity and quality.To that end, our continuous Integration and continuous deployment (CI/CD) pipelines, which are run on Google Cloud and Gitlab, always need to be at optimum performance. Over the years, we’ve built a lot of integrations and have automated testing and validation, which enables us to deploy with confidence. 

Expanding into new markets globally with Google Cloud

During late 2018 to early 2019, we decided that it was time to grow our presence globally and began looking for a platform that would support our ambitions. We were already on a hybrid cloud with different providers, which made sense in our earlier days as it provided more support in the APAC region. However, our goals were different as we looked to expand, and we found that Google Cloud was best suited to meet our needs because of its well-established global network. Google Cloud’s highly connected cloud infrastructure and advanced inter-region capabilities meant that we could easily build out into different geographies. 

Together with the scalability of Google Kubernetes Engine (GKE) and agility of BigQuery, our team has been able to take on projects that have become increasingly more complicated and challenging as we continue to extend our global network. 

Relying on the stability of Google Cloud for CI/CD

Airwallex relies on powerful and flexible APIs and CI/CD pipelines for seamless international money transfers, among other services. GitLab and GKE provide a scalable, always-available environment that enables us to respond to the rapidly changing security and market conditions. 

All our GitLab service components are run natively in GKE clusters. We use Cloud SQL, Google Cloud’s fully managed relational database as our GitLab database, and the cache service on Memorystore and Cloud Storage buckets act as our GitLab object storage. Some of our GitLab runners are also hosted on Compute Engine virtual machines. 

The diagram below illustrates how Google Cloud is integrated into our GitLab CI/CD pipelines:

A key benefit of Google Cloud is scalability. Running GitLab on GKE has provided us with immense benefits in terms of scalability and reliability. For example, the autoscaling of GKE and the Horizontal Port Autoscaling (HPA) feature of Kubernetes means we never have to worry about the load on GitLab. 

Kubernetes nodes can be autoscaled based on the node pressure, and the replicas can also be autoscaled based on the performance of containers. Our GKE clusters are deployed regionally across three zones, which improves the availability of our GitLab instance. Using Cloud SQL, we can guarantee a 99.95% SLA for our GitLab database, and we can deliver up to 99.999% availability by backing up the GitLab data on Cloud Storage buckets — building a truly global DevOps culture.

In the past, everything was manual, and it used to take us eight hours to deliver a release to production. With an automated CI/CD pipeline, we can deliver a change in less than an hour, including manual approvals from our stakeholders.

Identifying fraud efficiently with Vertex AI 

As a fintech company, data security is paramount. We need to ensure that we are working with verified institutions, which is why fraud detection is the first line of defense. In August 2023, we launched a new project using Vertex AI, focused on anti-money laundering and fraud analysis using generative AI technology. We implemented Vertex AI on our proprietary fraud detection system to scan any website that a customer gives us to determine its validity. After we verify that the website is legitimate, it is put through our other machine learning models to measure the risk of that website.

Prior to this, we relied on BigQuery to train our machine learning models to detect fraud on our system. With Vertex AI, we can detect fraud in real time, improving the speed and efficiency of our day-to-day operations and enabling us to react to potential fraud much quicker. However, we continue to use BigQuery to support other data-driven decision-making due to its minimal maintenance and high security. The serverless, highly scalable infrastructure of BigQuery makes it an incredibly economical tool. 

Delivering mission critical infrastructures with a trusted partner

For Airwallex, we collaborating with the Google Cloud Professional Services Organization (PSO) has been a game changer. The team’s willingness to share their in-depth knowledge on system designs and SRE practices inspire our engineers to continuously do their best work. Having a dedicated team to support us also gives us the confidence to experiment and develop new products quickly, knowing that our infrastructure is secure.

Tencent is a leading internet and technology company headquartered in Shenzhen, China. Our mission is “value for users, tech for good.” We are also the company behind world-famous games like Level Infinite, PubG Mobile, Honor of Kings, GTFO, and Assassin’s Creed Jade. In February 2020, we acquired Funcom, a gaming company that recently celebrated its 30th anniversary, with a critically acclaimed portfolio that includes games like The Longest Journey, Anarchy Online, and Metal Hellsinger.

The acquisition of Funcom brought together the best of tech and the gaming industry, and with that, we decided to build an analytics culture at Funcom, supported by Google Cloud. We’ll share the challenges we faced as well as the solutions that were implemented in the process.

To demonstrate, we’ll use the online multiplayer survivor game, Conan Exiles, as an example. Developed by Funcom and released in 2017, this game constantly updates and releases new content, and was transitioned into a live service game model in 2022. As such, we needed data to support our business decisions. 

An architecture for scalability and growth

Funcom’s architecture was developed to support the internal development team with live operations and to monitor the game servers’ health. The entire architecture was made of on-premises virtual machines and open-source frameworks, which limited our use cases and scalability. The legacy technology stack was not built with a data-driven approach in mind from a live service game model standpoint. 

Based on interest from both developers and executives at Funcom, we decided to collaborate with Google Cloud to develop a new architecture. With only a few months to go before the release of the first season of Conan Exiles, the Google Cloud team provided us with a fully operational data warehouse that could be used to build dashboards and provide insights to key stakeholders, including executives, marketing, and live operations. The diagram below illustrates the architecture that we used:

We built our new technology stack according to a few key criteria: ease of integration, diverse use case coverage, and optimizing the total cost of ownership (TCO).

Building this data platform was like putting together puzzle pieces. We replaced our legacy data infrastructure using key products like Cloud Storage and BigQuery, which acted as a data lake and query engine. As a result, we were able to build a robust data pipeline and a well-established data platform foundation in less than two months, enabling access to a host of new game data that was previously unseen, such as in-game player activity playtests. This includes marketing data, such as social listening and community responses, performance data about CPU, graphic or memory usage, and even crash monitoring data. 

A new foundation to connect gaming and marketing datasets

With the new architecture set up, we decided to explore other ways of using data to optimize cost performance and get better control of the data stack to connect marketing and sales datasets. For example, game leads need to understand how data can support in-game development, while our marketing teams should have easy access to in-game data to support marketing efforts.

To help, we automated the entire pipeline with always up-to-date KPI reports to monitor our marketing performance within Google Cloud. Additionally, the data team can provide recommendations based on in-depth insight analytics by connecting the data across player behavior, community, and marketing.

Moving forward with a revenue-generating tech stack 

With Google Cloud, we’ve been able to redesign the raw data pipeline and data lake architecture without impacting our gaming data pipeline, day-to-day decision-making systems, or requiring additional engineering overhead. As a result, we are able to process twice as much game data on a daily basis, compared to our previous architecture. In addition, we have also decreased our overall monthly costs by 70% using BigQuery and Cloud Composer.

Moving forward, we’re looking to further expand this architecture with near real-time pipelines built with Pub/Sub. We are also planning to improve data quality monitoring and alerting, and standardizing our data structure, so that we can deploy newly developed features directly to beta and accelerate our time to market.

Editor’s note: Today, we’ll hear from Trendyol, a leading ecommerce marketplace and Google Cloud partner based in Türkiye, about how it uses Looker on top of its BigQuery and BigLake data lake infrastructure to allow its thousands of users to gain real-time insights while maintaining strong governance in the cloud. 

The beauty of cloud environments is that rather than a “one size fits all” solution, you can choose the services you need and start using them immediately. Still, ensuring enterprise-wide best practices and compliance, such as ISO27001, can sometimes be a challenge, creating additional authorization requirements and friction between teams. This inherent conflict can lead to competing priorities between the groups consuming cloud resources and those responsible for looking out for the greater good of the organization. In fact, 69% of organizations indicate that stringent security guidelines and code review processes can slow developers significantly.

With this in mind, let’s take a look at how leading ecommerce platform Trendyol uses Looker to visualize petabytes of data ingested from hundreds of sources and achieve cloud governance at scale.

Balancing innovation and risk mitigation

Within an organization, it is rarely a single development team utilizing cloud services but a complex combination of various internal, production, and non-production applications, each vying for priority and with its own set of unique demands. Additionally, it is important to consider that various governance entities, including IT and organizational governance, as well as audit teams, may also be involved — and you need to solve for all parties.

Let’s explore the two different sides:

Freedom: Developers crave the freedom to experiment and innovate. They need to quickly prototype new ideas and bring them into production. If development cycles take longer, they are hesitant to make changes.

Governance: There needs to be governance to ensure systems are secure, reliable, and scalable. This includes enforcing coding standards, conducting security reviews, and monitoring system performance, especially for the resources shared across disparate teams.

Within cloud environments, these dynamics can prove more challenging. At Trendyol, for instance, thousands of users leverage Looker for everyday business intelligence tasks, including processing data and delivering reports that help optimize operations, personalize experiences, and predict market trends. While the majority are data analysts, there are many other types of users tapping into Looker across the company, including business users, warehouse operations, and cargo operations.

As one would expect, all of these users want to use data in their own way to speed up their priorities and make a difference in how they work, such as setting up recurring schedules to deliver data from Looker to Google Sheets. While these capabilities provide powerful ways to gain insights, it’s also important for Trendyol to ensure these actions are taken fairly and efficiently to avoid impacting experience or incurring unnecessary costs.

This is a common scenario for many organizations as they embrace cloud services. Cloud capabilities enable teams to outsource the heavy lifting and focus on more valuable business priorities. At the same time, they may not always be cautious or fully aware of risks, which can adversely impact the existing business — despite their good intentions.

Therefore, it’s critical to strike the right balance between innovation and risk mitigation to create systems that enable productivity while also being safe and reliable.

Creating trend-setting ecommerce experiences with Looker 

Trendyol aims to let users explore and discover new insights while keeping its platform fast, secure and affordable. This is not just a technology challenge, but one that requires collaboration among teams and big-picture thinking.

With Looker, Trendyol can centralize key metrics from its existing tools and platforms in near real time, empowering its data analysts, business users, and other teams with streamlined data analytics and insights to power their daily workflows. Moreover, Looker provides easy ways to implement data governance, including access control features, allowlists, job scheduling, cache optimizations, and more.To ensure cloud governance and maintain flexibility, Trendyol’s data warehousing team created a set of internal mechanisms and principles using Looker. This included the following:

Creating an official allowlist. Trendyol identified its top users and content to proactively create and maintain an allowlist of scheduled jobs. This allowlist is continuously monitored for new additions and regularly sanitized in collaboration with schedule owners to determine which jobs should remain. The cleanup process is automated, allowing Trendyol to eliminate duplicate schedules and enabling schedule owners to grant permissions to allow jobs to stay on the allowlist.

Establishing forums for user collaboration. Trendyol set up formalized programs for users to join and learn from one another, focusing on how top performers are utilizing scheduled jobs. This includes regular meetings, office hours, and other forums for discussions to encourage better collaboration. 

Spreading out scheduled job execution times. Updating and processing data are some of the most compute-intensive tasks in Looker. Trendyol regularly evaluates whether scheduled jobs actually require execution at peak times and explores alternatives, such as eliminating duplicate jobs, adjusting scheduled times, or removing old and unnecessary jobs.

Reviewing job frequency. Assess whether certain jobs need to be scheduled for frequent recurrence and consider reducing that rate to optimize resource utilization. For example, the team might assess whether scheduling a job once a week is sufficient to meet user needs rather than scheduling it multiple times a day. 

Addressing zombie schedules. If a schedule owner leaves the company, Trendyol determines whether another schedule owner can take over their scheduled jobs. If no other schedule owner can be found, these “zombie” schedules are removed from Looker.

Delivering a Hub and Spoke model and cache optimizations. Trendyol adopted a hub and spoke architecture model, which allows the company to apply universal business logic while allowing individual teams to define their own logic and access control rules and policies. Trendyol also makes use of Looker’s smart caching mechanisms to avoid fetching redundant data for similar queries.

All together, Trendyol has found that combining these efforts and continuously working to improve them has made it possible to provide true cloud governance at scale. Trendyol can now handle more than 260,000 queries a day during peak periods and empowers more than 1,500 employees to use Looker insights for their daily work.

To learn more about Trendyol’s journey on Google Cloud, see the following:

E-commerce data warehouse migration | Google Cloud Blog 

BigQuery and BigLake: Real-world data products for AI/ML at scale

Google Cloud Platform – Trendyol Tech – Medium 

2021 Trendyol Data Management Day

Trendyol Data Management Day 2022 

We also recommend checking out the following Google Cloud guides for more data governance best practices: 

Best practices for planning accounts and organizations 

Managing contacts for notifications 

Security Command Center best practices

Guide to Cloud Billing Resource Organization & Access Management

Most data analysis starts with exploration — finding the right dataset, understanding the data’s structure, identifying key patterns, and identifying the most valuable insights you want to extract. This step can be cumbersome and time-consuming, especially if you are working with a new dataset or if you are new to the team. 

To address this problem, we announced the preview of new data insights capability in BigQuery at Next ‘24 that surfaces relevant, executable queries for tables that you can run with just one click. These features are available as part of Gemini in BigQuery and leverage the metadata and profiling information of tables from Dataplex.

In this blog post, we explore how Alex, a data analyst working for a large enterprise, can use the new BigQuery data insights features to accelerate his analytics workflows. Like many data professionals, he often encounters the “cold-start” problem when exploring new datasets. With little or no prior knowledge about the data they’re working with, it can be difficult to identify patterns, much less uncover valuable insights. We also dive deeper into the concept of grounding generated queries and the roles of different personas involved in this journey.

Addressing the cold-start problem with data insights

Data insights harnesses Google’s Gemini models to generate insightful queries about hidden patterns within a table, utilizing the table’s metadata. By analyzing data types, statistical summaries, and other metadata attributes, it helps data analysts like Alex overcome the cold-start problem and unlock a world of data exploration possibilities.

“The Insights feature felt like it understood the table, it filtered out not so useful columns like Created_at time, Transaction Id, while highlighting important columns like Amount, Intent Type, Bank Name, App Version, Platform.” – Product Manager, Financial Services Industry

Grounding generated queries for data relevance and accuracy

One of the key features of BigQuery data insights is its ability to ground generated queries. This means that the queries are based on the actual data distribution and patterns within the dataset, ensuring their relevance and accuracy. The grounding process involves:

Analyzing profile scan data: Data insights examines the published profile scan data of the dataset, which includes information such as data types, statistical summaries, and other metadata attributes.

Generating queries based on data distribution: Using the profile scan data, data insights creates queries that are tailored to the specific data distribution and patterns within the dataset.

Validating queries: The generated queries are validated to ensure their relevance and accuracy.

Two key personas: admin and data consumer

Two primary personas who can benefit from using BigQuery data insights:

Admins – responsible for generating insights using the data insights feature. Admins typically include data steward, data governors, or other technical users who have the necessary permissions and access to the underlying data. 

Data consumers –  can view and execute the generated queries without needing direct access to the underlying data. Data consumers may include business analysts, data scientists, or other non-technical users who rely on the insights generated by BigQuery data insights to make informed decisions.In our story, Alex is a data consumer.

Getting started with BigQuery data insights

To use data Bigquery data insights, follow these steps:

Access data insights: With your data in BigQuery, navigate to the BigQuery Studio in the Google Cloud console. Here, you’ll find an overview of your tablesand their associated metadata.

Generate queries: Select a table and click on the “Generate insights” button. Data insights analyzes the metadata and generates a list of insightful queries tailored to your dataset.

Explore and refine queries: Review the generated queries and refine them as needed. 

Run queries: Execute the queries against your table and analyze the results to uncover valuable insights.

Alex’s path to greater data insights

Initially, Alex struggled to get up to speed when working with a new dataset. However, after discovering BigQuery data insights, he was able to streamline his data exploration process. Here’s what data insights brought to Alex’s work:

Efficient data exploration: By automatically generating insightful queries based on metadata, data insights enabled Alex to explore new tables more efficiently and independently.

Time and resource savings: With data insights handling low-to-moderate complexity data analysis tasks, Alex was able to focus on more challenging projects and save valuable time and resources.

Collaboration and democratization: Data insights made data analysis more accessible to non-technical users in Alex’s organization, fostering collaboration and promoting a unified approach to data interpretation.

Real-time insights: By automatically deriving insights from continuously flowing business data, data insights helped Alex and his team respond to changing business conditions in real-time.

“It’s fantastic that the Insights generation feature in BigQuery not only provides new insights but also simplifies the process of running derived queries. The tool surprises me with fresh perspectives, going beyond what I initially considered. Its user-friendly nature makes it accessible to everyone, enabling efficient query execution.” – Data Analyst, renewable energy industry

Unlock insights from your data, fast

BigQuery data insights is a powerful tool to help you unlock valuable insights from your data. By leveraging the metadata of tables, it streamlines the data exploration process and enables data professionals to focus on more challenging tasks. The grounding of generated queries ensures the relevance and accuracy of the insights, while the two primary personas – admin and data consumer – facilitate collaboration and democratization of data analysis. 

Check out the documentation to learn more about data insights and reimagine the way you explore and analyze data.

PostgreSQL is an advanced open-source, relational database management system that empowers businesses with robust, reliable, and scalable database solutions. It has a rich set of features including security, availability, and performance that ensure data integrity and business continuity. With its extensive extensibility and vibrant community support, PostgreSQL offers a flexible and cost-effective alternative to proprietary database systems, making it an ideal choice for organizations seeking a versatile database foundation for their critical applications.

AlloyDB for PostgreSQL, meanwhile, is Google Cloud’s fully managed, enterprise-grade database service that adds several enhancements to the standard PostgreSQL database engine while maintaining full compatibility: AlloyDB can handle the most demanding workloads with superior performance, high availability, and scalability, outperforming standard PostgreSQL. Further, AlloyDB leverages Google’s differentiated infrastructure and machine learning for automated database management, intelligent resource optimization, and accelerated analytical queries, making it an attractive choice for organizations that need a powerful and streamlined PostgreSQL-compatible solution. So, which should you choose?

At Google Cloud Next 24, we showcased AlloyDB provides up to 2x better price-performance than self-managed PostgreSQL and up to 4x better transactional performance on the same-sized instance. This blog dives deeper into the data behind that comparison, and explores additional value AlloyDB delivers.

Comparing performance

With AlloyDB, we made several enhancements to PostgreSQL’s kernel including transactional and query processing layers and added self-managing autopilot features such as automatic memory management and adaptive vacuum management. When coupled with its intelligent distributed storage system and automated storage tiering, AlloyDB performs far more efficiently than standard PostgreSQL. 

We carried out a performance comparison evaluation to contrast the cost-effectiveness of AlloyDB with self-managed PostgreSQL running on Google Compute Engine (GCE), both in terms of total cost of ownership (TCO), as well as cost per transaction. For our evaluation, we compared AlloyDB for PostgreSQL configured with HA on 8 vCPU / 64 GB Memory with self-managed PostgreSQL 15 running on two VMs configured for HA, each with 16 vCPU / 128GB Memory: 

The table and chart below shows the price/performance improvement:

Despite being configured with only 50% of self-managed compute resources, AlloyDB exhibited better TPM. With databases configured with 30% of the data cached, both performance and price-performance are up to 2x compared to self-managed standard PostgreSQL workloads. AlloyDB’s CPU utilization was close to 90%, suggesting the AlloyDB kernel is much better utilizing the available resources.

By deploying AlloyDB using the same GCE SKU (16 vCPU/128GB), we observed a 4x increase in performance. AlloyDB is engineered to scale with a larger number of vCPUs, resulting in better price-performance with a higher vCPU count. Currently, AlloyDB supports up to 128 vCPUs, which equates to 864GB of memory.

Standard PostgreSQL deployed on Compute Engine encounters resource limitations such as IOPS constraints and CPU scaling issues, hindering its ability to handle large amounts of data. Also note that this analysis does not include the manual resources required to self-manage the database, any additional inter-zone networking-related charges which can add to overall costs. 

Additional cost savings

AlloyDB compute instances in a cluster leverage the same regional colossus storage. What this means is that, when you deploy HA and add replica nodes (up to 20 at the time of publishing this blog), they all use the same storage. For example, if you have a 5TB database, and you have a HA + 4 read replicas, in a self-managed environment, the storage cost will be 6x the capacity (one primary, one standby, and 4 read nodes). With AlloyDB, you will be paying for 1 x 5TB costs.

Also, you don’t have to pre-provision AlloyDB’s storage capacity and allocate more capacity for IOPS. With AlloyDB’s cloud-scale architecture, the storage usage is dynamic which expands and also shrinks based on the usage. You pay only for the capacity you use.

There are no additional costs for AlloyDB cross-zone data replication for HA and for replica. With up to 25x lower read replica lag, you can scale horizontally using the same storage and with near-real time data.

Besides, AlloyDB can be a single datastore for your transactional, analytical and vector database workloads. You don’t have to provision multiple databases for these purposes which reduces additional compute, storage, networking, and ETL deployment and management aspects.

Managing PostgreSQL databases on your chosen infrastructure can offer cost advantages but comes with several challenges and hidden costs. These include capacity planning for peak workload, ongoing maintenance of hardware resources, and manual management of essential database tasks like deployment, scaling, security, high availability, backups, and disaster recovery. This can put a strain on your team and lead to higher administrative overhead and costs.

AlloyDB provides a fully managed experience that addresses these requirements and helps you run your mission-critical applications with a 99.99% uptime SLA, including maintenance.

AlloyDB comes out ahead

AlloyDB surpasses self-managed databases by offering up to 2x price-performance improvement and more cost savings with shared regional storage. Additionally, AlloyDB significantly mitigates the challenges associated with self-managed databases through automation of routine administrative tasks, on-demand vertical (read/write) and horizontal (read-only) scalability, minimal maintenance downtime, and inclusion of columnar engine & AI capabilities. With these features and benefits, AlloyDB provides a better, faster, and cheaper alternative to self-managed PostgreSQL environments.

Next steps

Watch Andi Gutmans’s Spotlight session from Google Cloud Next ‘24 showcasing various innovations with AlloyDB

Learn about AlloyDB at AlloyDB for PostgreSQL intelligent scalable storage

For a comprehensive overview of AlloyDB, refer to the overview section of the documentation.

Start building on Google Cloud with $300 in free credits and 20+ always free products at https://cloud.google.com/free