Cloud

To be “historic,” an event or place must be associated with a degree of importance that makes it worthy of notice, study, or preservation. 
To be “old” merely means something has been around a long time or occurred long ago. 

Regulations and compliance enforce companies notably Insurance, finance, healthcare to preserve their customer, applications and transaction data for a minimum period. The period can be as low as 7 years and as high as forever. As a result, organizations end up having a huge amount of archive data which is typically stored in physical or virtual tapes. The archive data stays inactive and becomes ‘old’ in no time unless the government agencies ask for past data or reports.

With increased adoption of cloud technologies, there are now innovative ways and means to transform the archive data from ‘old’ to ‘historic’. Google cloud and Model9 have partnered to transform legacy archive data into a valuable asset and means of differentiation.

Below are the few reasons why archive data transformation should be given a serious thought. 

Simplification: On-premises tape and virtual tape infrastructure can be complex to operate and maintain. Mainframe tape subsystems need specialized skills in a field where skills are a scarce resource as it is.  Not only do on-premises tape systems require specialized skills but they also require a proprietary FICON infrastructure that can not be shared with other platforms and that needs to be maintained and replaced every 5 years. In addition to the aforementioned shortcomings, physical tapes are also prone to failure requiring on-site personnel to replace physical media, restart failed backup jobs etc. A cloud based solution to backup and archive data removes all that complexity with the simplicity of an industry standard cloud access, freeing up critical skills and eliminating complex infrastructure operations and planning.  

Cost benefits: Cloud based Coldline and Nearline storage has reached incredibly competitive price points compared to virtual or physical tape libraries and their specialized FICON SAN infrastructure.  

MIPS Reduction: Model9 moves the data processing from General Purpose (GP) MIPS on the mainframe to zIIP processors thereby reducing GP MIPS consumption and hence lowering overall mainframe consumption and cost (monthly license charge or tailor fit charges)

Leaner Mainframe: Moving the archive data which typically runs in 100’s of terabytes to 10’s petabytes, to cloud storage makes overall mainframe estate much leaner and hence becomes manageable to move to cloud.

Data corruption/Cyber Resiliency: Backup copy on cloud can be used to restore mainframe applications in the event of any unforeseeable cyber attack or data corruption due to any software upgrade, data contention or any other reason. Google cloud can provide pristine backup anywhere in the globe. 

While the aforementioned use cases have a lot of intrinsic value, they are geared more towards keeping archive data without changing their value for an enterprise from ‘old’ to ‘historic’. The next set of use cases focus on extracting value from an organization’s archive data. Mainframes have long been the backbone of many organizations and oftentimes contain decades worth of data such as customer interactions, credit card transactions, airline bookings and policy data just to name a few. Unlocking this treasure trove of data for some of the below use cases  can yield tremendous value out of an asset that was previously thought of as a liability or technical debt.

Massive data sets to train algorithms: Google cloud is focused on building numerous industry specific use cases leveraging the power of Vertex AI to transform the business. Some examples are fraud detection, automated claim settlement, inventory management. Mainframe archive data can contain decades and 100’s of Terabytes of historical data can be used to better train and refine machine models.

New product/business model: Rising trends in Gen Z like Micro lending, social economy, everything-as-a-service are sprouting new business models and products. Archive data transposed with current socio-economic patterns can help validate the success and viability of new models before these are rolled out to a large market.  

Consumer behavior: Predictive analysis of historical data can provide valuable insights about the consumer behavior at individual, region and geographic levels which can help organizations take critical decisions about cross-selling and upselling along with decommissioning of slow moving catalog items. These optimizations will lead to a leaner catalog with lower shelf life  and higher margin.

Reporting: When authorities ask for data or reports from the archives, a typical turnaround is 4-5 weeks to identify the right dataset, restore the tape, run the jobs, create the report and then share. With archive data on Google cloud, commonly requested reports can be configured with Looker so the authorities request can be responded to in a timely manner which will foster a better public-private partnership.

Above use cases just scratch the surface of what can be done once Google’s BigQuery and Analytics tools start working their magic on the decades of historical data stored in mainframe archives. 

Historical data processing by Google’s BigQuery and smart analytics solutions enable limitless data-driven innovation to drive customer delight and business amplification.

On the data transformation roadmap, organizations can commence from moving archive data to Google cloud, then HSM storage and finally active DASD storage. Once a solid, consolidated and comprehensive data management and governance platform is established on Google cloud, focus may be shifted to either moving workload or building new applications using cloud native technologies on the available data on Google cloud.

This is part 2 of a three-part series of posts, in which we summarize Workflows and service orchestration patterns. You can apply these patterns to better take advantage of Workflows and service orchestration on Google Cloud.

In the first post, we introduced some general tips and tricks, as well as patterns for event-driven orchestrations, parallel steps, and connectors. This second post covers more advanced patterns.  

Let’s dive in!

Design for resiliency with retries and the saga pattern

It’s easy to put together a workflow that chains a series of services,  especially if you assume that those services will never fail. This is a common distributed systems fallacy, however, because of course a service will fail at some point. The workflow step calling that service will fail, and then the whole workflow will fail. This is not what you want to see in a resilient architecture. Thankfully, Workflows has building blocks to handle both transient and permanent service failures. 

In our post on Implementing the saga pattern in Workflows (and its associated e-commerce sample), we talked about how you can apply the saga pattern and take compensation steps to undo one or more previous steps with the try/except block for permanent service failures. 

We also showed how to handle transient service failures by adding the default HTTP retry policy with the try/retry block of Workflows. If you have non-idempotent steps, you need to adjust the retry policy to default_retry_non_idempotent. In most cases, you need a custom retry policy with a longer backoff, because the default one waits at most ~8 seconds.  When eventual consistency over a longer time period is more important than fast failure, consider a longer retry policy over several minutes or even hours with a large multiplier is much more likely to succeed during temporary outages. 

Don’t take network calls for granted. Make sure you design your orchestration with resiliency in mind using retries and the saga pattern.

Wait for HTTP and event callbacks instead of polling

Sometimes, your orchestration might need to wait for a long-running job or an asynchronous event in another system before it can continue with the rest of the steps. The workflow can ask for that input by polling an endpoint or a queue, but this requires complicated polling logic, wasted polling cycles and most likely higher latency. A better approach is to use Workflows callbacks to wait for HTTP calls or events.  

In our Introducing Workflows callbacks post, we showed a workflow that waits for human input for automated machine learning translations. Similarly, the Smarter applications with Document AI, Workflows and Cloud Functions post shows a document processing workflow that waits for human approval of expense reports with a callback. 

While both of these posts are focused on waiting for HTTP callbacks, in Creating Workflows that pause and wait for events post, we showed how a workflow can also wait for Pub/Sub and Cloud Storage events. You can even use Google Sheets as a quick and simple frontend for human approvals as we showed in the Workflows that pause and wait for human approvals from Google Sheets post. 

When designing a workflow, consider waiting for HTTP calls and events, rather than polling.  

Orchestrate long-running batch jobs

If you need to execute long-running jobs, Google Cloud has services such as Batch and Cloud Run jobs that can help. While these services are great for completing  long-running jobs on Compute Engine instances and containers, you still need to create and manage the Batch and Cloud Run job service. One pattern that works really well is to use Workflows to manage these services running batch jobs. 

In the Taking screenshots of web pages with Cloud Run jobs, Workflows, and Eventarc post, we showed how Cloud Run jobs take screenshots of web pages and Workflows creates and manages parallel Cloud Run jobs tasks. Similarly, in the Batch – prime number generator sample, we showed how to run prime number generator containers in parallel on Compute Engine instances using Google Batch. The lifecycle of the Batch job is managed by Workflows.

Use the right services for long-running batch jobs and use Workflows to manage their life cycles.

Treat serverful workloads as serverless with Workflows

Sometimes, you really need a server due to some serverless limitation. For example, you might need to run on GPUs or execute a long-running process that lasts weeks or months. In those cases, Compute Engine can provide you with customized virtual machines (VM), but you’re stuck with managing those VMs yourself. 

In this kind of IT automation scenario, you can use Workflows to create VMs with the customizations you need, run the workloads for however long you need (Workflows executions can last up to one year), and return the result in the end.  This pattern enables you to use servers but manage them as if they were serverless services using Workflows.

In our Long-running containers with Workflows and Compute Engine post, we showed how to use Workflows to spin up a VM, start a prime number generator on the VM, run it for however long you want, and return the result.

Next time you need to spin up a VM, treat it like a serverless service with Workflows. 

Run command-line tools with Workflows and Cloud Build

We often use command-line tools such as gcloud to manage Google Cloud resources or kubectl to manage Kubernetes clusters. Wouldn’t it be nice if we could call these tools from Workflows and orchestrate management of resources?

In the Executing commands (gcloud, kubectl) from Workflows post, we showed how to use Cloud Build to run these tools and how to create and call that Cloud Build step from Workflows using the Cloud Build connector. 

Keep in mind that this pattern is not limited to gcloud and kubectl. Any tool you can run in a container can potentially be a target for Workflows with the help of Cloud Build.

Integrate command-line tools into your workflows when needed by calling a Cloud Build step from Workflows.

This second part series covered a lot of ground, but there’s still more to cover! We’ll wrap up the series in our third and final post, which describes how to manage the lifecycle of workflow definitions and the benefits of using Firestore. Stay tuned! For questions or feedback, feel free to reach out to us on Twitter @meteatamel and @glaforge.