Cloud

More and more organizations are running data applications on Google Kubernetes Engine (GKE), as evidenced by the number of Kubernetes clusters running stateful applications on GKE, which has been growing exponentially since 2019, doubling every year on average. And with the rise of both AI/ML workloads and data on Kubernetes, customers are looking for more integrated solutions across compute and storage. Specifically, customers were looking for ways to access AI/ML data stored in Google Cloud Storage (GCS) to be made easily available to containers on GKE using file semantics.

Cloud Storage FUSE for machine learning

Cloud Storage is a common choice for AI/ML workloads looking to store and access training data, models, and checkpoints. With Cloud Storage FUSE, objects in Cloud Storage buckets can be accessed as files mounted as a local file system, providing a frictionless experience for applications that need file system semantics. Cloud Storage FUSE is available today in Public Preview, with official Google support. You can deploy Cloud Storage FUSE as a regular Linux package, as part of integrations with Vertex AI, and as of today, as part of a managed turnkey offering with GKE.

Cloud Storage FUSE CSI support on GKE

The new Cloud Storage FUSE Container Storage Interface (CSI) driver on GKE available in public preview allows Kubernetes applications to mount Cloud Storage buckets as local file systems.  This provides:

Portability: Mount and access Cloud Storage buckets with standard file system semantics, providing portability for ML workloads that eliminates application refactoring costs. Use the CSI driver to operate with familiar kubernetes APIs. 

Massive scale: From training, to inference, to checkpointing, you can leverage the performance of Cloud Storage to run your ML workloads at scale.

Streaming data support: Start training jobs quickly by providing compute resources with direct access to data in Cloud Storage, rather than having to manage logic to copy it down to a local filesystem instance. This means you don’t need to wait for data to download before doing meaningful work. 

Built-in support for GKE Standard and Autopilot: The CSI driver deploys Cloud Storage FUSE under the covers without a user needing to install or manage it. The Cloud Storage FUSE CSI driver on your cluster turns on automatic deployment and driver management. The driver works on both Standard and Autopilot clusters.

Non-privileged access: Previously, you needed to create your own solution to run Cloud Storage FUSE on GKE, which required privileged access. The Cloud Storage FUSE CSI driver does not need privileged access, enabling a better security posture.

Authentication out of the box: You can use Workload Identity to easily manage authentication while having granular control over how your Pods access Cloud Storage objects.

Extensive support for accelerators: The Cloud Storage FUSE CSI is supported on all accelerators available on GKE including GPUs and TPUs.

Use cases 

For workloads that require file-system semantics, Cloud Storage FUSE CSI support enables: 

ML training (using Pytorch and Tensorflow ) on GKE, including reading data and checkpointing saved models using Cloud Storage as the source of truth

ML inference models that infer results from files stored in Cloud Storage 

Write and read back checkpoints and saved models to/from Cloud Storage 

Accelerated startup time of your data and AI/ML applications by streaming data dynamically 

Python-based third-party data apps where customers don’t have control over source code

Usage considerations  

When evaluating Cloud Storage FUSE, please be advised of the following:

Cloud Storage FUSE and its CSI driver are not alternatives for a fully managed file system such as Filestore. For example, it does not provide concurrency control for multiple writes to the same file, is not fully POSIX compliant, and does not support NFS/CIFS/SMB. Use Filestore with Multi-shares for use cases where multiple writers need to write to the same file, or you need true POSIX support. See here for more information on usage patterns with Cloud Storage Fuse.

The CSI driver is only supported starting with GKE version 1.26. 

Terraform is not supported as part of the Public Preview but will be supported for GA. 

Official support is available for PyTorch and Tensorflow ML frameworks, while other workloads receive best effort support.

Getting started 

To get started on GKE, simply enable the Cloud Storage Fuse CSI driver on GKE and authenticate (via Workload Identity) with the Cloud Storage bucket that you would like to mount as information in your Pod specification. Learn more here. 

To use Cloud Storage FUSE on a VM, please download the Linux package. Learn more here.

A few years back, it would’ve taken more than half-a-day for an official of the Utah Division of Wildlife Resources (DWR), part of the Utah Department of Natural Resources (DNR), to track a Rocky Mountain elk. The process involved hauling a very high frequency (VFH) antenna to several locations until it picked up the signal from the elk’s radio collar. The official would then triangulate the animal’s approximate location by picking up the signal from multiple locations surrounding the elk. Given the complexity and manual labor involved, Utah DWR could track a single collared animal only once in a fortnight, or even a month. 

Fast forward to 2023, and now Utah DWR officials can get real-time location data for thousands of animals without leaving their office. What’s more, they can overlay today’s location data with up to 20 years of historical information to make data-driven decisions to improve wildlife habitat and populations using the state’s Wildlife Tracker application, running on Google Cloud. With this application, DWR can better gauge how wildlife interacts with the landscape over time, identify new migration routes, assess areas that need habitat improvements, or determine where to put fences or crossing structures to keep animals from crossing roads in areas with high-collision risks. These alerts on the movements of animals over time are also enabling real-time situational analysis, such as how wildfires or drought impact populations and migration patterns. 

According to Jessie Shapiro, GIS project manager for the Utah DWR, “We are now able to know and understand our wildlife so much better. We understand patterns of migration, causes of death, and can obtain individual data points for thousands of animals. It has opened our eyes to how the animals are using the landscape, and what interventions we need for their safety and protection.” 

Singular queries, less worries

Before moving to Google Cloud, Utah DWR officials said they had very little insight into the behavior of the state’s wildlife. For instance, by the time officials reached the site of the corpse of a dead animal with a radio collar, scavengers had already gotten to it, making it nearly impossible to determine the cause of death. As Shapiro said, “We were not realizing the true potential of what the data was telling, and we needed to change that.”

The backend legacy system processing the GPS data from thousands of animals radio collars was also riddled with inefficiencies. Queries often led to complete system crashes, and it took days to get historical data in a format that was easy to understand and act upon. The staff wanted these tools, but were frustrated with the inability to use them efficiently. With the database expected to surpass 20 million GPS locations, modernizing the system was critical for the department to scale and make data-driven decisions. 

In 2021, Google Public Sector helped Utah DWR rebuild their Wildlife Tracker application on Google Cloud. The aim of the project was to create a serverless data warehouse that helped them to scale rapidly, and take away the concerns regarding limited resources, hardware and other management tools. With BigQuery, a completely serverless, enterprise data warehouse solution, DWR officials can quickly search their database of historical data to identify animal behavioral patterns. A cloud-based solution also makes its data warehouse infinitely scalable without having to worry about hardware upgrades as its dataset continues to grow at pace. 

Setting a new standard

Since the implementation went live in August 2020, Google Cloud has helped DWR eliminate lag from their radio collar tracking system. Now, DWR officials can search over 20 years of historical data with a single query in minutes rather than days. They are also able to better understand and utilize their data to determine data-based population management, evaluate migration and the effect of climate patterns, and better track wildlife births and mortality.

As for tracking that Rocky Mountain elk today, all it takes is one query and under a minute to zero in on its exact location, see where it’s been, and determine its pattern of movement.  

DWR’s pioneering work is leading to new data-driven approaches to support its mission to help sustain, enhance, and conserve wildlife in Utah for generations to come.

Over the past few years, breakthroughs in computer vision and video analytics have generated a lot of interest – and put attention on Developers who work with large amounts of video assets. Unfortunately, many of the benefits of these breakthroughs in computer vision have remained elusive. Vertex AI Vision makes it easy for developers to build, deploy and manage computer vision applications. 

Challenges of building compelling computer vision applications have been plentiful, but two big buckets have emerged: data engineering and data science. On the data engineering side of the house, ingesting large amounts of video into a system has proven time consuming, expensive, and error prone. Furthermore, storing that data and then making use of the insights from that data requires advanced skills in data warehousing, search, analytics etc.

On the data science side of the house, creating custom image classification, object recognition, and image segmentation requires sophisticated machine learning skills that are often hard to find. Effective, highly tuned models for emerging use cases such as occupancy analytics, PPE detection, and vehicle detection require even more specialized skills. 

Vertex AI Vision

Vertex AI Vision solves these challenges by providing an end-to-end application development environment that makes it easy for developers to: 

Ingest camera video feeds in real time 

Analyze the video using advanced AI pre-built or custom models 

Store, search and retrieve video data at a massive scale

In this blog, we’ll walk you through Vertex AI Vision and show you how to get started. As always, our AI products also adhere to our AI Principles.

What developers need to know

A Vertex AI Vision application consists of these three main components:

Streams

Warehouse

Processors

The Stream component connects to the camera and streams data into the application. You can think of it as a source where data is getting sent in from the cameras or video files.

The Warehouse component is a repository of video and metadata attached to the videos. You can think of the warehouse as the sink of the video stream. You can additionally search the warehouse for clips on the metadata associated with the video clip, like the time duration, number of vehicles etc. This can be done both via UI and via a comprehensive set of API.

The Processor component is a function that takes the input from the source (Streams), transforms them and sends them to a sink (Warehouse or BigQuery). The processors are typically Computer Vision Models that can perform machine learning tasks on the video frames like Object Detection, Vehicle Detectors, Person blur, Occupancy count etc. You can additionally chain processors before storing the output in a warehouse or BigQuery. You can also attach your own processors by building a custom model with Vertex AI AutoML Object Detection or with Vertex AI custom modeling. Each of the processing nodes have input and output streams. Vertex AI Vision provides APIs to read the intermediary output of the streams.

Vertex AI Vision also provides the capability to store the outputs of the Processors in a BigQuery Table. This can be used to build in Looker or Data Studio dashboards.

An application is an abstraction that ties the Sources, Sinks and the Processors together in a nice Graphical User Interface.

In the sample application shown in Fig 1, the stream is connected to a pre-built Object Detector model (processor), the output of it is stored in Vision AI Warehouse and BigQuery. The object detector can identify more than 500 types of objects. The processor model infers on the video frames and generates metadata which is sent to the Vision AI Warehouse and BigQuery. 

Connecting the cameras to the Streams in Vertex AI Vision is easily done via a command line tool called vaictl or a C++ API.

The video ingestion_time (the time the video stream reaches the video stream from the remote cameras) field is used as a key to correlate metadata and frames across the app.

The pre-built models include:

People/Vehicle Detector

Person Blur

Object Detector

PPE Detector

We also provide an advanced analytical processor called Occupancy Analytics processors, which not only detects people and vehicles it also sends the count information over to the sink (warehouse and BigQuery).

We additionally provide a model to filter out inactive video frames from being sent to the applications.

Steps to Build an Application include:

Creating a Stream

Creating a Warehouse

Creating an Application

In the application UI

Drag the Stream component and register the above stream

Attach the right processor for your stream

Store the output of the processor in Warehouse, BigQuery or both

Deploy the application

Resources for developers to get started

As a developer, you want to write great code that is both useful and is used. You want to use the tools you are familiar with already and you want to leverage other assets (code, libraries, frameworks etc) wherever it makes sense. In the case of video analytics you want to focus on what you do best, building great apps, and not on the vagaries of data engineering or the complexities of model building, tuning etc.

For detailed steps follow the Vertex AI Vision docs. The security prerequisites are listed here. Once the application is deployed you can start streaming your camera stream or a video file using the vaictl command. The syntax of command is provided in the documentation. 

Once you start the stream you can view the stream inside the application and search for videos clips in the vision warehouse. Have fun!

Important resources for you to get started with Vertex AI Vision include:

Tutorials

Create a face blur app with warehouse storage

Create an occupancy count app with remote streaming

Create occupancy analytics app with BigQuery forecasting

Documentation

Vertex AI Vision Overview

Vertex Vision AI Quickstart

Ingest Streams

Create a Warehouse

Videos and Demos

Next 2022 launch video

Free tier/ Cloud Credits

Google Cloud Credits

Google Cloud Free Tier

In recent years, machine learning (ML) has gained tremendous popularity as a powerful tool for solving complex problems across various domains. However, building and deploying ML models at scale can be challenging, as it involves a range of tasks such as data preparation, feature engineering, model training, deployment, monitoring, maintenance and so on. 

According to “The Art of AI maturity” report published by Accenture “87% of data science projects never make it into production.” This is where MLOps comes in – it can help to address the core challenges by providing a framework for managing the entire ML lifecycle, from data collection and preparation to model development, testing, and deployment. It also reduces the time from ML model development to production and increases the success rate of ML projects.

In Google Cloud, we understand how important MLOps is to successfully productionize ML models. So we collaborated with Equinix, the world’s digital infrastructure company™ and  a leader in global colocation data center market share, with 248 data centers in 27 countries on five continents. We helped them by providing the advisory services on the MLOps best practices and architecture. 

Let’s take a sneak peek at the MLOps requirements at Equinix and the final architecture that was proposed.

What does MLOps mean for Equinix?

After multiple discovery sessions with the Equinix Team, we identified the core requirements and pain points to address in their new MLOps architecture:

Reusability: Components such as features and pipeline components should be reused across projects to reduce costs and improve efficiency.

Foundations: The foundations of the infrastructure, such as environments, folder structure, and project hierarchy, should be well-designed to support scalability and reliability.

Early identification of problems: Problems should be identified early by including data validation, notifications, and retry mechanisms.

Cost optimization: Costs should be optimized by paying only for what is used.

Enterprise CI/CD requirements: Enterprise CI/CD requirements should be met by integrating with GitHub and GitActions.

Scaling: The infrastructure must scale to support future growth.

Security: Enterprise security requirements should be met in terms of IAM roles, network, etc.

MLOps Architecture

Based on the above requirements from Equinix, key design considerations were made for example – using Vertex AI Feature Store instead of Big Query for online feature serving, using DataFlow for pre-processing vs using the existing python based pre-processing and so on. After carefully assessing all the alternatives, the below reference architecture for MLOps in GCP was proposed:

Reference architecture for MLOPs using GCP is illustrated in Figure 1. This architecture includes the following pipeline stages:

Vertex AI Workbench enables data scientists to quickly explore new ideas and develop/experiment new models. The source code is saved in GitHub repository

Github Actions with self-hosted runners are integrated with Github as the source code repository. This enables continuous integration of code, quality and security scans. The artifacts generated during the process are saved in Artifact Registry and Google Cloud Storage. These artifacts are deployed to implement the pipeline.

Unit tests and integration tests can be performed during the continuous integration in Github Actions with self-hosted runners. End-to-end tests are performed on demand in the continuous integration pipeline. 

Metadata about the artifacts is generated and saved in Vertex ML Metadata.

Automated triggers can be enabled to run pipelines. For example, one trigger is the availability of new training data. These triggers can run the model training pipeline and the new trained model can be pushed to Vertex AI Model Registry. 

To train a new ML model with new data, the deployed Vertex AI Pipeline is executed. 

To train a new ML model with new implementation, a new pipeline will be deployed through CI/CD pipeline.

“The proposed architecture design covers the requirements and scenarios that we were looking for. As our AI and ML portfolio is growing in scale and complexity, it’s important to follow a clear and up-to-date architecture if we want to keep increasing the value delivered by our solutions. As part of the process, the team also acquired the skills required to fully implement it” according to Bernardo Fernandes, Data Science Senior Manager at Equinix.

Below is the snapshot of the features before and after MLOps implementation at Equinix:

As businesses increasingly rely on machine learning to gain a competitive edge, MLOps has become a critical component of their strategy. By adopting MLOps practices, organizations can achieve faster time-to-market, better performance, and higher ROI for their machine learning initiatives.

Fast track end-to-end deployment with Google Cloud AI Services (AIS)

The partnership between Google Cloud and Equinix is just one of the latest examples of how we’re providing AI-powered solutions to solve complex problems to help organizations drive the desired outcomes. To learn more about Google Cloud’s AI services, visit our AI & ML Products page.

We’d like to give special thanks to Nitin Aggarwal, Vijay Surampudi, Parag Mhatre and Anantha Narayanan Krishnamurthy for their support and guidance throughout the project. We are also grateful to the super awesome collaboration with the Equinix Team (Ravi Pasula and Brendan Coffey, Bernardo Fernandes, Łukasz Murawski, Jakub Michałowski, Sonia Przygocka-Groszyk, Marek Opechowski, Daria Bondara, Nila Velu, Vijay Narayanan, Dharmendra Kumar, Shailesh Sukare, Arunraj Kumar Raje, Seng Cheong Lee).

Imagine you just got the restaurant bill and you can see only one number: the total cost, tip included. You have a sense you overpaid. Also what about this significant delay with service? Is it reflected in the tip? It’s difficult to answer these questions if you don’t have visibility into a breakdown of the costs. 

This is very similar to the situation many market participants find themselves in the electronic trading industry. Transaction Cost Analysis (“TCA”) has been focusing for many years in the total “bill” of a trade or some components of it (e.g. bid-ask spreads), but hasn’t been providing visibility into other aspects such as market impact (e.g. did our trading volume impact the prices?), delay costs (also called “slippage”, remember this service delay at the restaurant?) and opportunity costs (maybe I should go for my second favorite dish if I know I have to wait too long for my first choice). 

BidFX, a global FX multi-dealer platform collaborating with Google Cloud is trying to change this. “For a long time, everyone involved in the execution of financial assets has wanted to know how badly/well they’re trading. This has been done often in a non-differentiating way and is called TCA, whether to meet regulatory requirements or to add value by improving decision making. What we’re doing with our latest analytics is breaking that down even further to get to the real source of the costs being incurred. These tools work in facilitating a good marketplace for all participants.” says Daniel Chambers, Head of Data & Analytics at BidFX.

Working with Google Cloud, BidFX has been able to develop and deploy a new product called Liquidity Provision Analytics (“LPA”), launching to production within roughly six months, to solve exactly this industry challenge in an innovative way. LPA will be offering features such as skew detection for liquidity providers, execution time optimization, pricing comparison, top of book analysis and feedback to counterparties. 

“For BidFX this is an important product as it places BidFX ahead of the curve in its industry. It is a type of analytics that many users within the industry will not currently be doing, but will add a lot of value to our clients.” mentions Chambers.

Leveraging Google Cloud’s data and analytics capabilities to tackle a complex business problem 

The challenge: storing and serving FX time-series data in real time

BidFX’s LPA needed to store FX tick data and derived analytics at various resolutions (e.g. 1m, 1h, 1d). Each price data feed is defined by several fields: 

a client ID,

a timestamp,

the resolution (e.g. 1 min),

the deal type (e.g. forward, swap etc.), 

the currency pair (e.g. GBPUSD), 

the liquidity provider (i.e. the origin of the quote), 

a tenor (i.e. the settlement date) and,

the quantity to be traded. 

Any results of slicing and dicing, aggregating or probing the analytics derived from these data feeds are served to BidFX’s clients globally through a front-end application. The main latency requirement of the platform was to be able to serve end-clients under 500 ms. 

How did BidFX solve the challenge?

Building on Cloud Bigtable, Google Cloud’s fully managed, enterprise-grade NoSQL database service, BidFX’s LPA product can auto-scale seamlessly with no administrative or maintenance effort. The solution was able to achieve a back-end to front-end latency of ~35ms @50th and ~340ms @99th in the same geographic region, with availability of 99.999%. 

““Leveraging the elasticity and operational efficiencies of Google Cloud allows us to reduce our time to market, delivering innovative, web-scale products in a lean manner while being confident we don’t encounter bottlenecks that require re-architecting at critical moments.”, says Maksim Korneichik, Engineering Manager at BidFX.

Bigtable in a nutshell

Bigtable is essentially a sparsely populated table that can scale to billions of rows and thousands of columns, enabling the storage of terabytes or even petabytes of data. Bigtable separates architecturally compute (i.e. nodes) and storage (i.e. “SSTable” files in Colossus, Google Cloud’s distributed file system), to achieve up to single-digit millisecond latency (within a region), essentially limitless scale, and 99.999% availability, while scaling directly proportional to the number of machines added to the cluster. To achieve this performance, the schema needs to be designed and optimized to the patterns of usage.

LPA schema and benchmarking considerations for BidFX

Bigtable is a key/value store and each table has only one index, the row key, with rows sorted lexicographically by row key. Choosing and structuring the key is critical in attaining the best read and write performance and to avoid hotspots. For time-series datatwo design patterns are useful: a) short and wide tables, where rows are time buckets and new events are added as new cells or new columns, or b) tall and slim tables where rows represent single timestamps and events are added as new rows.

For the LPA product most of the queries start with a particular client in a particular time window and for particular deal types. There are six main dimensions in the queries: the client ID, the liquidity provider ID, the currency pair, the deal type, the quantity and a timestamp for the time bucket. In any single query, five out of these six components are kept constant while the search space spans across the remaining dimension. For instance, client A might be looking to retrieve all bid/ask quotes (aggregated e.g. at per second resolution), for all currency pairs, from liquidity provider B, for spot and for quantity X, at time bucket T. In this example, the varying component is the currency pair. Similarly, an LPA client might be trying to retrieve all quotes from all liquidity providers for a particular quintuple of client ID, currency pair, deal type, quantity and time bucket, in which case the varying component is the liquidity provider ID.

To reduce the read latency, one “bid/ask” column family was created to contain the quotes as data in each cell and all the data are replicated four times in a space-time tradeoff. Each replica of the data is structured to have a different format for the row key, thus enabling the different types of queries at the required low latency. The four different types of keys are:

clientID_liquidityProvider_dealType_quantity_timestamp,

clientID_currencyPair_dealType_quantity_timestamp,

clientID_currencyPair_liquidityProvider_quantity_timestamp,

clientID_currencyPair_liquidityProvider_dealType_timestamp

The query component that is left out of the key in each case is incorporated as a column qualifier to index the required information and separate tables are created for aggregated data at different resolutions (e.g. aggregated data per second, minute, hour etc). One example of this approach is illustrated on Table 1.

To benchmark the performance of the application 1,000 queries with a collection of various Client IDs were used, each one of which hit at maximum 28,800 rows*. This is expected to be a worst case scenario limit as typically clients hit fewer rows per query. The Key Visualizer was used to monitor hotspots and bottlenecks in querying performance, while the Java BigTable client with channel refreshing was used to implement the benchmark tests. 

Looking into the future of electronic trading infrastructure: how can Google Cloud help?

A typical electronic trading platform today

Looking beyond client and trade analytics, on a typical electronic trading platform (e.g. Figure 2), a common user journey for a trader is as follows: the trader interacts with a web application where they can see an aggregated book of liquidity and/or they can request quotes (RFQs) for specific instruments. The trader then can place various types of orders (fill-or-kill, limit, good-till-canceled etc.) or use execution algorithms (TWAP, VWAP etc.) to express their market convictions. The order is usually being managed by an order management system, abstracting the interactions with the underlying protocol communications (e.g. FIX), and is executed by an execution engine which interacts with the markets (e.g. other banks, non-banks, broker-dealers etc.). Finally, the trader receives one or more “fills” for the parts of the order that were fulfilled in the market, if the order was at least partially filled.

Behind the scenes there are a lot of other systems. A pricing engine that generates the RFQs in a smart way that is profitable for the liquidity provider and appealing to the end user, a time-series database that records every market “tick”, and various trade ledgers, e.g. for risk management, settlements, and reconciliations that potentially need global synchronization capabilities.

The challenge with architecting such a platform

Now, imagine you are specifying out the requirements for such a system to be deployed on-prem. What are you provisioning for? Peak volumes at e.g. non-farm payroll days? A trading activity multiple, e.g. 3-5x of the peak volume of a historical period? In any case, you either end up with a very expensive platform that is not used at full capacity or with a cost effective but unreliable platform when customers need it most, thus losing immediate and long term business when volumes peak. Eventually without continual management, you will end up in both of these situations, an expensive platform that isn’t really ready for sudden and new peak volumes.

Distinguishing between the latency-critical and non-critical path

Engineers of trading platforms usually differentiate between the latency “critical” path where components of the platform tend to stay in a colocation facility, where market participants share data centers with common length wiring, and the “non-critical” path.

The majority of the components comprising a trading platform are not in the critical path and they make excellent candidates for cloud-native applications, leveraging the elasticity and operational efficiencies of cloud.  

Five additional ways Google Cloud can help electronic trading platforms to scale and innovate faster, securely and more cost effectively

Faster time-to-value for Quantitative Research: Typically components such as the pricing engine are powered by models developed by a quantitative research (“QR”) team. Oftentimes though QR teams are bogged down by siloed data and legacy model development tooling that doesn’t scale and restricts model deployment to production only to batch mode, instead of being able to capture and monetise intraday changes of the market. Trading platforms can significantly accelerate the time to market and value by using analysis-ready market and reference data as a service through BigQuery, or by sharing data securely and privately within the organization through Analytics Hub. Ultimately QR can move to a cloud-native, fully-managed and scalable modus operandi of data acquisition & discovery, burst compute workloads for model calibration, repeatable, auditable and scalable model development and AI/ML tooling such as NLP APIs and document understanding.

Fully managed, reliable and globally consistent trade and risk ledgers: For instance, Cloud Spanner, a fully managed, mission-critical, relational database service that offers transactional consistency at global scale, is a unique solution for a globally available and synchronized ledger to store and subsequently reconcile and settle trades. Spanner evolved from internal Google applied research and is battle-tested by Google’s own products and services used by billions of users. 

More reliable, faster and secure market data connectivity and delivery channels: Today, cloud is becoming pervasive in market data distribution, with 93% of exchanges, trading systems and data providers offering cloud-based data and services, according to a Google Cloud commissioned survey. Google Cloud’s global scale, low latency and highly reliable network allows for real-time market data delivery and consumption, while Global Load Balancer options enable trading platforms to direct orders to preferred liquidity venues in a scalable and fully managed manner. Additionally, Datacast, a market data transcoder, is bridging the gap between trading protocols (e.g. binary FIX, ITCH, etc) and modern formats (e.g. Avro) that are optimized for throughput and latency at the expense of readability and interactivity with modern systems.

Streamlined risk reporting and back office operations: Google Cloud’s Regulatory Reporting Platform offers a cost-efficient, reliable and fast way to facilitate near-real-time or interactive reporting and it can enable an operating model where near-real time risk management is possible and traders can run what-if scenarios in a few seconds or minutes before they place orders. 

Regulatory compliance, resilience and security: Infrastructure-as-Code (IaC) and Risk-and-Compliance-as-Code (RCaC) can significantly reduce time to deploy new compliant venues (think… liquidity pools out-of-the-box), and migrating or running workloads on Google Cloud goes a long way towards strengthening operational resilience towards the path of preparing to address regulatory frameworks such as the Digital Operational Resilience Act.

Special thanks with equal contribution to this blog post to Svitlana Gavrylova, Data Practice Lead UK&I; Alison Rooney, Global FSI Partner Manager; Matthew McGibbon, FSI Customer Engineer, and Salvatore Sferazza.

*The maximum number of data points (rows) per instrument considered for benchmarking was one day’s worth of minute data, i.e. 24h*60 min = 1,440 rows and for a maximum of 20 instruments per query we have 20*1,440 = 28,800 rows per query.

When you are looking to modernize your organization’s tech stack or build a new transformational application, your choice of database becomes important. 

There are many factors including cost, security, scaling characteristics, self-managed vs. fully managed, on-prem vs. public cloud, etc., that go into the decision-making process. 

Cost is one of the key considerations for businesses while selecting a database. So, how do you figure out the “true cost” of using a particular database? 

In this blog, we go through a simple framework for evaluating the total cost of ownership (TCO) of a database technology, which will help you in the decision making process.

Five types of costs

At a high level, we can bucket all the costs associated with managing a database into 5 categories

Take the hypothetical example of an organization that wants to self-manage a database cluster. For the purposes of this exercise, we’ll define a database cluster as a setup that has more than one node (for e.g., a self-managed sharded MySQL cluster). Most scenarios also apply to self-managed databases in the public cloud.

1. Hardware and licensing costs

On-prem databases require upfront investments in hardware such as servers and storage disks, apart from indirect costs associated with planning, purchasing, and deployment of hardware. 

Initial costs such as services needed to install, configure and tune the database software also add to the upfront costs.

A clustered/sharded database adds more complexity to this process, particularly when hardware and software components are from different vendors.

Self-managed databases in the public cloud also require provisioning of server machines and storage disks based on the peak workload demands. So, you will incur higher costs even when the databases have less load.

You will have to pay licensing fees for some database technologies (e.g. Microsoft SQL Server licensing fees).

2. Operational costs and efficiency 

Operating and maintaining the database: Operating the database takes up a non-trivial amount of time and resources. You will need to plan and execute operating system and database version upgrades, testing before upgrades, continuous tuning and optimization, and security patches.

Depending on your database size and the database’s availability SLO, the cost can be quite high. For example, a company self-managing their scaled and sharded MySQL database may end up spending significant resources (in terms of both dollar and people costs) in operations and maintenance. 

Scaling up and scaling out: If you are self managing a database, how do you plan for special events like Black Friday, New Year’s Eve, the Superbowl, or new product launches? If you overprovision the hardware, that’s an unnecessary cost during non-peak periods. Also, overprovisioning hardware means overprovisioned licenses. If you underprovision, you’re at a risk of missing business opportunities. If your product becomes an overnight success (e.g., a mobile game launch), you have to worry about your database not keeping up with demand. When your database hits a vertical scaling limit (e.g., machine size), you have to scale out using sharding, which means you incur additional costs including rearchitecting, making changes to your app, operational costs and possible downtime. Even if re-architecture is not required, non linear scaling remains a concern. 

Data resiliency: To prevent data loss and ensure business continuity during server failures, network failures, or natural disasters such as heat waves or hurricanes, you need to plan for data resiliency by setting up data replication across data centers in physically separated locations/regions. There’s a significant cost associated with this setup (hardware, software licensing costs, and potentially the licensing for the replication software), as well as an ongoing maintenance burden. 

Geographic expansion: As your business grows to different geographical locations, you may need to replicate your database for reasons including data residency requirements and to reduce database interaction latency for clients/applications. Setting up replication, achieving data synchronization between replicas, avoiding split brain issues, and handling replica recovery from failures introduces significant operational costs.

Efficiency: Certain SaaS application providers (who are cloud customers as well) provision one database per customer. Managing a large number of customers translates into managing a large number of individual databases. This will cause inefficiencies, leading to higher operating costs.

3. People costs

As database maintenance tasks and database management complexity increase, you will need more experts to maintain your databases. Sometimes, these costs could be much higher than your overall infrastructure (hardware) costs.

4. Cost of outages

Direct revenue loss: This one is obvious — if your business is not up and running, it can cause a direct revenue loss. Even a partial database outage can cause revenue loss.

Operational toil: In case of a database-related outage, how much toil does it cause to your organization? How many resources does it take to be on call, respond, mitigate, conduct incident post-mortems and put in preventive strategies?

Contracts: Some business-critical workloads may have contractual and regulatory repercussions for outages or application downtime (e.g., stock exchanges, banks).

Brand trust issues: If your business is down due to database maintenance windows or database outages (low availability), you risk your brand being associated with lower reliability, and lower availability.

Opportunity costs: Are you spending your resources just managing databases and the associated infrastructure rather than leveraging them to enhance and build new capabilities to support the core business?  

5. Productivity costs

Time to production: Do you have the right CI/CD pipeline to optimize your time to production? Do you need to set up multiple dev, pre-production and staging environments with different database instances as a path to production? What is the time and cost associated with maintaining multiple environments and a non-optimized (database-dependent) time to push a change to production? How much effort do you need to spend planning and preparing for applying schema changes?

Poor performance: Have you optimized your database for best application performance? A poorly tuned database can translate into poor application performance and wasted resources.

Slower application development: Does your database enable your organization to develop and launch applications quickly? Do you have a local copy of the database that you can download on developers’ machines (or laptops) for rapid development? Does the database have a familiar interface for faster development? 

Increased Complexity: Do your developers have to spend significant effort working around the complexities of your database deployment? Do you need to allocate your rockstar developers to solving database scaling / reliability challenges instead of core business differentiators?

Calculating business benefits

When developing applications, ask yourself whether your database allows you to iterate quickly and push changes to the production environment quickly? The time it takes to deploy code to production will directly translate into how fast you can launch new applications or features and how fast you can innovate.

While we took the hypothetical example of a company running their own MySQL cluster, the list above applies to many other situations as well. When selecting a database, it is important to look at the total cost of ownership, or TCO. To solidify the points above, let’s look at one of our managed database services, Cloud Spanner, and its TCO.

Introduction to Cloud Spanner

Cloud Spanner is a distributed, globally scalable, strongly consistent relational database service built specifically for the cloud. It not only offers SQL benefits with relational database structure, but also offers horizontal scalability typically offered by non-relational databases — so that users get the best of both. This combination is designed to deliver high-performance transactions and strong consistency across rows, data centers, regions, and continents with an up to 99.999% availability SLA and enterprise-grade security. Users can focus on building applications and delivering innovation, while Spanner fully manages database infrastructure, sharding, replication, and failovers, with no scheduled downtime. 

Spanner TCO

An ESG study on the economic benefits of Cloud Spanner concluded that hosting your data onCloud Spanner is 78% less expensive than using on-premises servers and up to 37% less expensive than using other cloud databases.

Customers across different industries such as financial services, healthcare, retail, gaming, telecommunication, and technology are reducing their database TCO by migrating to Cloud Spanner. For example, ShareChat reduced the costs of its non-relational workload by 30% by moving over 120 tables with 17 indexes from AWS DynamoDB to Spanner. Learn how another customer, Kochava, realized significant savings by consolidating their relational workload running on hundreds of MySQL databases into fewer autoscaled Spanner instances. 

Total database costs with Cloud Spanner

Let’s look at typical database costs when you run with Cloud Spanner.

Hardware and licensing
No need to over provision hardware or pay hefty licensing fees to use Spanner. You can simply pay-as-you-go for the database compute capacity and the storage consumed in Spanner. You can use Spanner autoscaler to flexibly scale up and scale down database compute capacity which drastically reduces the cost.

Operating and people costs
As a fully managed cloud database service by Google Cloud, Spanner lets you spend less time managing, and more time building applications and adding value to your customers. It is very easy to create backups and to restore; there’s no patching or upgrading required. You can roll out schema changes without downtime, and security is built-in.

Efficiency 
Spanner allows you to consolidate several self managed databases into one instance and manage it efficiently.

Node costs
Spanner offers a Free Trial for 90 days. Spanner also has a downloadable Emulator, which can be used for correctness testing and development, completely free. Start for as low as $65 per month, and purchase Committed Use Discounts to further lower your cost by up to 40%.

Scaling
Start with 100 Processing Units (Smallest size) and go up to an unlimited number of nodes. Spanner can be scaled on demand without downtime as the application grows, so there is no need for over-provisioning to reserve capacity ahead of time. 

Customers have reported that Cloud Spanner enabled them to enhance business agility, which allowed them to harness the value of their data to better service existing customers and open new revenue streams. Spanner gives you complete flexibility to have your database in a single region or multiple regions spanning the globe, with the ability to do stale reads or strongly consistent reads.

Security
Spanner offers, by default, encryption for data-in-transit via its client libraries, and for data at rest using Google-managed encryption keys. CMEK support for Spanner allows you to have complete control of the encryption key. You can also protect database backups with CMEK. Spanner provides VPC Service Controls support and has compliance certifications and necessary approvals so that it can be used for workloads requiring ISO 27001, 27017, 27018, PCI DSS, SOC1|2|3, HIPAA and FedRamp.

Availability
Spanner multi-region configuration offers 99.999% availability SLA with no maintenance windows or planned downtime. In case of a zonal or regional failure, recovery is automatic (no intervention needed) with 0-RPO (no data loss) and no loss in business continuity.

Replication
All data is automatically replicated to all the replicas in spanner and is included in the cost of a node.

Consistency
Spanner offers global external consistency guarantees. In traditional RDBMS environments, developers build application logic to handle transactions and/or eventual consistency. Because Spanner offers strong consistency, scale, and relational semantics including transactions, it can make it easier for developers to produce cleaner code. 

Transform your applications with a lower TCO 

In short, organizations rely on complex database management systems to ensure access to data and insights that they depend on to run their business operations. However, managing and maintaining these databases can be challenging from a total cost of ownership (TCO) perspective. Customers across industries such as financial services, retail, gaming, and healthcare are choosing Cloud Spanner databases to modernize and transform their applications at lower TCO. Get started with Cloud Spanner at no cost with Free Trial instances.

Every hour of every day, Google moves a lot of data around the world. But how? With the Effingo (Latin for ‘to duplicate’ or ‘copy’) data copying service, a Google-only service that moves data internally and on Google Cloud customers’ behalf.

As a Google Cloud customer perhaps you move data with Storage Transfer Service, a managed transfer option that provides out-of-the-box security, reliability, and performance, that eliminates the need to optimize and maintain scripts, and handle retries. Storage Transfer Service is useful for consolidating data from separate projects, to move data into a backup location, or to change the location of your data.

Likewise, Google’s internal services need very similar functionality to manage their data. This includes data that is in charge of the ecosystem of Google Cloud services, such as BigQuery and Cloud Spanner.

Effingo solves a challenging problem of data movement at a global scale. It supports a wide range of workloads with different traffic characteristics and different requirements for data replication, durability, and latency. In this article we will explore the main motivations behind data movement and solutions to core infrastructure problems that we face when copying exabytes of data on a daily basis.

Why move data in the first place?

But first, let’s think about why you even want to move or geographically distribute your data.

For one thing, you want to replicate your data for durability and reliability. You definitely don’t want to keep a single copy of your data. In the end, we’re talking about a file that is stored on a hard drive. Any hard drive can fail at any point. For this reason, keeping copies of your data in different locations helps ensure that at any point of time there is a copy of your file that you can read from. Moreover, these locations should be geographically distant from each other to avoid even temporary data loss due to natural disasters, network cuts, and other incidents that can impact a local area.

Second, you want your data to be close to your users to reduce latency. You want to serve data with minimal delay to give a great user experience. In many cases, delays caused by data transfers from remote locations can spoil the usability of your application or create the perception that your service doesn’t work. According to Google research, 53% of mobile users abandon sites that take over 3 seconds to load. There are other consequences of lack of data proximity: for example, transferring data from distant locations wastes network resources, which can have a negative impact on the environment. Google has been carbon-neutral for our operations since 2007, and we have matched 100% of the electricity consumption of our operations with renewable energy purchases annually since 2017, but we aim higher: Our goal is to run all our data centers and offices on carbon-free energy, 24/7, by 2030.

The third reason to geographically distribute your data is to balance storage capacity between clusters. This reduces the cost of storing data because it enables us to increase the overall capacity of our data centers and use computation power in less loaded clusters. We can then limit the amount of wasted resources and use otherwise idle hardware. This is especially relevant for batch processing and storing cold data where the exact storage location is not that important.

What are the challenges?

Operating at such a large scale as Google makes data movement a challenging problem for several reasons. Here are a few examples.

Data must be secure and consistent at its destination. Achieving this is complex. For example, to copy a file from one location to another, you need the following: throughput to read and write to disks, network capacity to transfer bytes between data centers, and compute resources to instrument the whole operation. 

Considering the volume of data that Google operates on, high and predictable throughput is one of the key features of a copy service at scale. It is typical to transfer terabytes of data per second, and so it’s essential to have a scalable service that can handle traffic with sometimes rapidly changing patterns. It poses a further question: how to run such a service and still be resource efficient?

Finally, it’s crucial to ensure high system availability. Google infrastructure is dynamic and copying data between any two clusters may behave differently at different points of time. Effingo needs to be resilient to cluster turndowns, network bottlenecks, rapid changes in resource demands and capacity, and so on.

How do we solve (some) of these problems?

One of the core use cases for data movement is replication. If you want to ensure that your data is reliable, you want to keep more than one copy, ideally in different locations, to be less prone to data loss.

Let’s consider the following example. You have a file in cluster A, and you want to make copies in destinations B – G.

In the simplistic approach, you would start copying to all destinations in parallel.

Unfortunately, it is highly inefficient to replicate your file this way. This solution doesn’t scale and requires significant investment in infrastructure. 

If you consider the cost of network infrastructure, moving data across the ocean is an expensive operation. Effingo creates data transfer plans that reduce the volume of data to transfer across the ocean. If possible, it reuses already replicated data as a new source.

To create data transfer plans Effingo needs to know alternative data sources and be aware of the network topology. 

If it goes to the alternative sources, Effingo stores a recent history of transferred files together with their metadata and time-bound permissions to access the files. Whenever a new copy is issued, Effingo checks whether there were copies of the original file that could serve as the same source but in the alternative location. Worth noting is that Effingo is very strict in verifying that both files — the requested source file and the alternative transfer file — are indeed the same to follow high security standards. Effingo not only checks whether the same user issued a copy from the same source but also if file properties including checksum, ciphertext checksum, mtime and several others match.

Once Effingo knows the source files and their alternative sources, it creates a transfer plan over the Google network. Effingo has a model of the network in the form of a graph where each location is a node and each edge is a weighted link between each node. For each copy Effingo creates a Minimum Spanning Tree over such a graph, which serves as an input to the transfer engine. Thanks to this approach we can select a plan that is optimal for each copy.  

In the next example, we show a more efficient approach. Effingo first makes an expensive copy to a remote location and then uses it as a new source. For instance, Effingo first copies a file from the US (source A) to Europe (destination C) and then uses the Europe-based file as a copy source for copies on the same continent. Note that Effingo never stores files in temporary locations for optimization purposes – the service only uses locations explicitly requested by our users.

In addition to resource efficiency, the main challenge is to ensure high throughput of data transfer. We solve this problem by applying several approaches that address different problems that may arise.

As described above, our customers’ workloads exhibit a range of different traffic patterns, which results in dynamic changes in capacity on each network link. For this reason, Effingo not only needs to respond to altering resource availability, but also to ensure user fairness in such conditions. In other words, Effingo wants all copy operations to progress while using resources in the best possible way. To achieve this, it uses sophisticated parallelism controls. These controls scale the service up to meet increased demand and limit service capacity if there is service overload, performance degradation, or resource waste.

Further, Effingo must be very resilient and react quickly to errors. On the file level, the service extensively uses retries for transient failures and aborts transfers quickly if it detects non-retryable errors. Effingo uses metrics, logs, and other observability signals to adjust data transfers when needed. For example, it can detect that copies from a specific source are slow and there is another file replica available. Effingo then reconfigures the copy operation to use the other replica to complete the data move.

Data movement at global scale is hard

Hopefully at this point you have a better understanding of why moving data is a challenging problem for infrastructure that operates at Google’s global scale. Effingo supports a range of services that run on Google infrastructure, including Google Cloud services and many internal Google services. While moving data at large scale requires a lot of attention to resource usage and resilience to support high throughput, there is good news: We keep working on this problem and make our infrastructure better every day, so you can run your business on Google Cloud and all data movement is transparent to you.