Cloud

Over the past several weeks, you’ve seen a collection of the most common questions and misconceptions the Cloud Functions Support team sees about Cloud Functions. Here is a recap of these most common “anti-patterns” and what you should do instead:

How to write event-driven Cloud Functions properly by coding with idempotency in mind: We explored what idempotent functions are and how this design pattern is important for background-triggered functions.

How to reuse Cloud Function instances for future invocations: We covered what global scope is for a Cloud Function, when to use, and what issues to look out for when used incorrectly.

How to establish outbound connections correctly:  We explained how to make outbound connections correctly by using explicit timeouts on your outbound calls. 

How to handle Promises correctly in your Node.js Cloud Function: We shared how to verify that you are handling promises correctly in Node.js to avoid saving undefined or cached data in your database.

How to run background processes correctly in Python: We cautioned why you should avoid running background tasks in Cloud Functions and explained why we recommend using Cloud Tasks instead.

We hope you’ve found this series helpful! Please let us know if you have any questions or ideas about best practices using Cloud Functions. You can reach us at @GoogleCloudTech on Twitter.

We recently announced a new goal of equipping more than 40 million people with Google Cloud skills. To help achieve this goal, we’re hosting Cloud Learn from Dec. 8-9 (for those in Europe, the Middle East, or Africa, the event will be from Dec. 9-10 and for those in Japan, you can access the event here), a no-cost digital training event for developers, IT professionals, and data practitioners at all career levels. The interactive event will have live technical demos, Q&As, career development workshops, and more covering everything from Google Cloud fundamentals to certification prep.

Here’s a more in-depth look at what to expect from Cloud Learn:

Hear from Google Cloud executives and customers 

Thomas Kurian, Google Cloud’s CEO, and I will kick off the first day by discussing how you can uplevel your career. The second day will begin with technical leaders from Twitter, Lloyds Banking Group, and Ingka Group Digital speaking with John Jester, our vice president of customer experience, about the impact of Google Cloud training and certifications they’ve seen in their organizations. 

Afterwards, you can choose from role-based tracks and join the training sessions most relevant to you. 

Training for developers 

For developers, Kubernetes expert Kaslin Fields will be guiding you through the following trainings during the first day: Introduction to Building with Kubernetes, Create and Configure Google Kubernetes Engine (GKE) Clusters, Deploy and Scale in Kubernetes, and Securing GKE for Your Google Cloud Platform Access. 

Google customer engineers Murriel Perez McCabe and Jay Smith will discuss how to prepare for the Google Cloud Professional Cloud Developer and Professional Cloud DevOps Engineer certifications on the second day. Jay will also walk you through a live demo of how to build a serverless app that creates PDF files with Cloud Run. 

Carter Morgan, a Google Cloud developer advocate, will end the second day with a session on actionable strategies for managing imposter syndrome in tech. 

Learning opportunities for IT professionals

IT professionals will have the opportunity on day one to learn from Jasen Baker, a technical trainer, how to get started with Google Cloud. Jasen will walk you through how to execute compute, store, and secure your data as well deploy and monitor applications. 

On the second day, you can hear from Google Cloud Certified Fellow Konrad Clapa and Cori Peele, a Google Cloud customer engineer, about how to prepare for Google Cloud’s Associate Cloud Engineer and Professional Cloud Architect certifications.  

Google Cloud experts will also take you through a live demo of how to create virtual machines that run different operating systems using the Google Cloud Console and the gcloud command line. Day two will conclude with a discussion from leadership consultant, Selena Rezvani, on how to negotiate for yourself at work, and speak up for what you want and need.

Training sessions for data practitioners 

Lak Lakshmanan, Google Cloud’s analytics and AI solutions director, and product manager Leigha Jarett will show you how to use BigQuery, Cloud SQL, and Spark to dive into recommendation and prediction systems on the first day. They’ll also teach you how to use real time dashboards and derive insights using machine learning. 

Author Dan Sullivan and Google Cloud learning portfolio manager Doug Kelly will begin the second day with a discussion on how to earn Google Cloud’s Professional Data Engineer and Professional Machine Learning Engineer certifications. You’ll also learn how Google Cloud Video Intelligence makes videos searchable and discoverable by extracting metadata with an easy to use REST API through a live demo on day two. 

Cross cultural business speaker Jessica Chen will end the last day with actionable communication tips and techniques to lead in a virtual and hybrid world.

Register here to save your virtual seat at Cloud Learn.

Predictive analytics has long been a source of strategic advantage for innovative companies. High-performing firms use strong predictive analytics to predict future outcomes more accurately, plan for unknown events, and discover product and customer opportunities.

One challenge with predictive analytics is producing the historical data to power the predictions. Some of the most important questions – like how COVID-19 will affect supply chains, or the effect of new technologies – often have little data to draw from, which makes it difficult to train machine learning models.

Today, predictive analytics customers fill the gap by relying on the judgment of their employees, partners, and vendors. So Google asked: how can we do this more rigorously? What if our collective wisdom could be combined with machine learning in a predictive model?

In 2020, facing the uncertainty of the pandemic, a group of Googlers launched an internal prediction market on Google Cloud to try this. Google first ran a prediction market from 2005 to 2007 that showed promise, and today’s team had two advantages we didn’t have in 2005: many more people at Google to participate, and the ability to build the platform on Google Cloud.

After iterating on the structure and design, and analyzing over 150,000 predictions from ~10,000 Google employees, we’d like to share the design patterns and technologies that helped to generate accurate forecasts on challenging domains like COVID-19, engineering milestones, and emerging technology trends.

How to Structure a Prediction Market

The core insight behind prediction markets is to incentivize the right people to forecast accurately, thereby producing a consensus forecast that is more accurate than any individual.

For example, let’s say you’re trying to predict: will interest rates rise in 2022? You may have ML models to forecast customer behavior and even predict macroeconomic trends, but anticipating events like this requires drawing on knowledge and judgments in your organization.

Accurately predicting the future with questions like this can be very difficult. On the one hand, it is tempting to be overconfident on a simple answer; on the other, it is easy to lament that one cannot possibly know. In fact, evidence from prediction markets and forecasting tournaments indicate that, with the right structure, organizations can achieve remarkable accuracy.

From our experience, we have arrived at five key principles behind soliciting accurate forecasts from your organization. Let’s walk through them step by step.

1. Provide a UI that makes it easy for people to express nuanced, precise predictions.

Most questions can be posed as forecasting a probability (as in the below example), date (“when will interest rates rise?”), or number (“what will the interest rate be next year?”). For each type, we give our participants two options: (a) with one click, predict that the probability/date/number should be lower or higher than the current consensus; or (b) provide their specific forecast as a range.

For example, consider the following probability market:

We recommend paying particular attention to the precise resolution condition, to ensure that everyone is forecasting the same thing. Precise forecasting can only work on precisely defined questions. Nominate admins from your org to review new question proposals, and ensure the right choice of source data and time frame.

2. Incentivize experts from around your organization to make predictions on questions where they have insights.

Like other prediction markets, our incentive system is similar to an options market. We convert predictions into “bids” and “asks”, interpreting the probabilities, numbers, or dates as “prices” and the confidence as “volume”. The application broker matches opposing bids and asks at the best price. The trade results in a contracts between the parties about the outcome, as shown in the following pseudocode:

In the same way that financial markets discover consensus prices by efficiently matching orders, prediction markets discover consensus forecasts by efficiently matching predictions. This maximizes the incentive for predictors to spot opportunities to correct the market by making a better prediction. When the question is resolved by updates in the real world, for each traded contract, one party will gain, and both parties can learn from the experience.

3. Support continuous predicting, so that the consensus forecasts update in real time as new information surfaces.

A particular challenge for predictive analytics is that new information can surface at any time. The stream of predictions, interpreted as bids and asks, represents a consensus of all participants as time passes. By graphing the bid-ask spread – the values for which, at that moment, no party wishes to take the other side – you can see the evolution of this consensus.

4. Give your predictors feedback on their performance, to help them improve their forecast accuracy over time. 

Once a question is resolved, in addition to notifying your users of the result, you can contextualize their performance: how well are they doing in that category, or in the current quarter? How do they compare to the top forecasters?

One incentive we offer successful predictors – beyond leaderboards and badges – is the ability to have a bigger influence on the market consensus by increasing the confidence of their predictions. Over time, predictors gravitate towards topics where they predict more accurately, which can also increase the market’s forecast accuracy.

Forecasting is like most other skills we use in the workplace, in that we improve over time. If incentives and feedback are structured correctly, expect improvements in accuracy both at the individual level and in the overall market. One key here is to ask questions over shorter time horizons. Instead of asking “what will our new service adoption rate be next year,” start by predicting the adoption rate next quarter, and iterate.

5. Synthesize the market forecasts with machine learning

Prediction markets fill the gap when there isn’t sufficient historical data to rely on machine learning alone. Most often, there is some data to inform your strategic decisions, but not enough to determine the right choice. For example, perhaps you have ample time-series data on your consumer spending behavior on one platform, but nothing yet on a new platform you have yet to launch.

There are two ways to synthesize prediction markets and machine learning. First, use the output of the prediction market as input into a larger predictive analytics system that also draws from machine-learned inference on time series data. Alternatively, add machine learning systems as market makers, providing baseline bid-ask spreads and liquidity that increases the incentive for accurate predictions.

Architecting the Platform on Google Cloud

By leveraging Google Cloud Platform (GCP), we were able to build the entire platform with a small team in less than a year and scale to thousands of users. We’d like to share the specific GCP tools we used to prototype, build, and grow a robust platform very quickly:

Serverless provided the main workhorse for the platform. We use App Engine, and are also experimenting with Cloud Run and Cloud Functions to implement the trading backend.

Serverless is a great toolkit for building an application before you decide what type of backends you might want to manage yourself on Compute Engine. It  handles deployments, versioning, and resource scaling without writing additional code. This lets us stay flexible with the backend architecture.

Cloud Scheduler and Cloud Tasks complement a serverless architecture well, making it easy to run periodic jobs, like providing badges to our top predictors, and long-running background tasks, like market resolutions and calculating analytics.

Cloud Firestore served as our application database, running in Datastore mode. As a scalable NoSQL cloud database, it lets you develop your frontend flexibly, updating your schema as you design and build more features. It also supports complex, hierarchical data – in our case, storing bids, asks, and traded positions as children of the market entity makes querying for data fast and easy. 

Using Data Studio, you can dashboard insights from the predictions, for example to visualize trends across markets over time. With Looker, you can aggregate the market predictions with existing predictive analytics to produce better insights than either would produce alone.  

It also supports an easy export to BigQuery for analytics and BigQueryML for Machine Learning, and to refine the human consensus algorithmically. Once you have this data, GCP’s Vertex AI offering can allow you to train forecasting models with little to no code, giving the best synthesis of human and machine reasoning on difficult strategic questions.

Dataflow enables us to do large batch operations on the data store – such as restoring from a backup – as a managed process.

We also use a variety of basic GCP tools for the operational side of running our prediction market. We used:

Logs Explorer to debug tricky situations, such as when queries time out or we hit an issue with an internal API

Cloud Profiler and Cloud Trace tools helped us identify scaling bottlenecks early, such as unnecessary synchronous Cloud Firestore queries that could be done asynchronously.

Cloud Error Reporting made it easy to configure monitoring for errors, jump to the underlying issues, and determine which errors were more more frequent

Get Started

Ready to take your predictive analytics to the next level and unlock the wisdom of your organization? If you’d like to discuss building a platform like this with us, please provide your information in this form.

We’re excited to announce that Datastream, Google Cloud’s serverless change data capture (CDC) and replication service, is now generally available. Datastream allows you to synchronize data across disparate databases, storage systems, and applications reliably and with minimal latency to support real-time analytics, database replication, and event-driven architectures. You can easily and seamlessly deliver change streams from Oracle and MySQL databases into Google Cloud services such as BigQuery, Cloud SQL, Google Cloud Storage and Cloud Spanner, saving time and resources and ensuring your data is accurate and up to date. Get started with Datastream today.

Since our public preview launch earlier this year, we’ve seen Datastream used across a variety of industries, by customers such as Chess.com, Cogeco, Schnuck Markets, and MuchBetter. This early adoption strengthens the message we’ve been hearing from customers about the demand for change data capture to provide replication and streaming capabilities for real-time analytics and business operations. 

MuchBetter is a multi-award-winning e-wallet app, providing a truly secure and enjoyable banking alternative for customers all over the world. Working with Google Cloud Premier Partner Datatonic, they’re leveraging Datastream to replicate real-time data from MySQL OLTP databases into a BigQuery data warehouse to power their analytics needs. According to Andrew McBrearty, Head of Technology at MuchBetter, “from MuchBetter’s point of view, leveraging Dataflow, BigQuery and Looker has unlocked additional insights from our ever-increasing operational data. Using Datastream in our solution ensured continued real-time capability – we now have trend analysis in place, improved efficiency across the business, and the ability to use our data to derive actionable insights and to make data-driven decisions. This means we can continue to grow and adapt at a pace our customers have come to expect from MuchBetter. And for the first time, the world of ML and AI is open to us.”

Getting to know Datastream

Google Cloud customers are choosing Datastream for real-time change data capture because of its differentiated approach:

Simple experience
Real-time replication of change data shouldn’t be complicated: database preparation documentation, secure connectivity setup, and stream validation should be built right into the flow. Datastream delivers on this experience, as MuchBetter discovered during their evaluation of the product. “Datastream’s ease-of-use and immediate availability (serverless) meant we could start our evaluation and immediately see results”, says Mark Venables, Principal Data Engineer at MuchBetter. “For us, this meant getting rid of the considerable pre-work needed to align proof of concept tests with third-party CDC suppliers.”

End-to-end solution
Building pipelines to replicate changes from your source database shouldn’t take up all of your team’s time. Use pre-built Dataflow templates to easily replicate data into BigQuery, Cloud Spanner or Cloud SQL. Out of the box, these Dataflow templates will automatically create the tables and update the data at the destination, taking care of any out-of-order or duplicate events, and providing error resolution capabilities. Leverage the templates’ flexibility to fine-tune Dataflow to fit your specific needs. “Google-managed Dataflow templates meant getting our pipelines up and running with minimal effort and fuss – this allowed more time to be spent on more complex pipeline development whilst tactically delivering solutions to our users,” says Venables.

Secure 
Datastream keeps your migrated data secure, supporting private connectivity between source and destination databases. “Establishing connectivity is often viewed as hard. Datastream surprised us with its ease of use & setup, even in more secure modes,” says Grzegorz Dlugolecki, Principal Cloud Architect at Chess.com, a leading online chess community and mobile application, hosting more than ten million chess games every day. “Datastream’s private connectivity configuration allowed us to easily create a private connection between our source and the destination, and ensure our data is safe and secure.”

High throughput, low latency
With Datastream’s serverless architecture, you don’t need to worry about provisioning, managing machines, or scaling up resources to meet fluctuations in data throughput. Datastream guarantees high performance – a single stream can process 10’s of MBs per second, while ensuring minimal latency. “We evaluated several market-leading ETL solutions”, says Dlugolecki,  “Datastream was the only tool able to successfully sync our complex, single-table datasets, doing this in weeks instead of years estimated by the other vendors.”

Getting started with Datastream

You can start streaming real-time changes from your Oracle and MySQL databases today using Datastream:

Navigate to the Datastream area of your Google Cloud console, under Big Data, and click Create Stream.

Choose the source database type, and see what actions you need to take to set up your source.

Create your source connection profile, which can later be used for additional streams.

Define how you want to connect your source.

Create and configure your destination connection profile.

Validate your stream and make sure the test was successful. Start the stream when you’re ready.

Once the stream is started, Datastream will backfill historical data and will continuously replicate new changes as they happen. 

Learn more and start using Datastream today

Datastream is now generally available for Oracle and MySQL sources. Datastream supports sources both on-premises and in the cloud, and captures historical data and changes into Cloud Storage. Integrations with Cloud Data Fusion and Cloud Dataflow (our data integration and stream processing products, respectively) replicate changes to other Google Cloud destinations, including: BigQuery, Cloud Spanner, and Cloud SQL.

For more information, head on over to the Datastream documentation, see our step-by-step Datastream + Dataflow to BigQuery tutorial, or start training with this Datastream Qwiklab.