Cloud

Every day, teams across Google Cloud come together to help address complex and pressing compliance, risk, and privacy requirements that enable customers to accelerate their digital transformation and innovate faster. Let’s take a look at some of our top trust and compliance-related efforts from the last few months: 

Combining data processing terms. In September, we updated and consolidated our data processing terms for Google Cloud, Google Workspace (including Workspace for Education), and Cloud Identity (when purchased separately). The combined terms are now one Cloud Data Processing Addendum (the “CDPA”). The updated terms maintain the previous benefits while strengthening our data processing commitments. We’ve also incorporated the new international data transfer addendum issued by the U.K. Information Commissioner. Learn more in our updated whitepaper. 

Seven Trusted Cloud announcements from Google Cloud Next ‘22. In October, we hosted Google Cloud Next ‘22, a global, 24-hour livestream and in-person event. At Next ‘22, we had seven new Trusted Cloud announcements, including Confidential Space, the next solution in our Confidential Computing portfolio. Confidential Space allows multiple parties to securely collaborate on joint tasks such as data analysis and machine learning (ML) model training. Read all of our Trusted Cloud announcements here. 

Advancing Digital Sovereignty. Also announced at Next ‘22 was our updated portfolio of Sovereign Solutions that help customers address their digital sovereignty concerns. These Solutions align with our 2021 “Cloud. On Europe’s Terms” announcement, and include Sovereign Controls, Supervised Cloud, and Hosted Cloud options. 

Expanding our cloud regions. In recent months, we announced five new Cloud regions, including Swedenand Norway, to support our customers’ digital transformation and sustainability efforts. These new regions will help our customers maintain security, data residency, and compliance standards, including region-specific data storage requirements. Keep up to date with our latest Cloud locations here. 

Validating records retention in Japan. In January 2022, the Japanese government’s revisions of the Electronic Records Retention Law came into effect. To address these changes, the Japan Image and Information Management Association (JIIMA) created a certification for software which complies with the new legal requirements. JIIMA has certified Google Workspace as compliant with this certification that will help our customers meet their regulatory requirements for digital record-keeping in Japan. 

Meeting additional requirements. In other compliance news, our latest U.S. Public Sector compliance authorization, StateRAMP, enables us to support government customers with enhanced data residency and support capabilities via Assured Workloads. In Germany, we updated our Bundesanstalt für Finanzdienstleistungsaufsicht (BaFin) offerings, and with support from our trust partner, microfin, we mapped our compliance posture to German requirements for IT (BAIT/VAIT) in banking and insurance. 

The completed Korea Financial Security Institute (K-FSI) audit. To help support a group of leading South Korean financial institution customers, we worked with auditors from K-FSI to complete an audit based on their guidelines. Our customers joined this “pooled audit,” which is an excellent example of industry collaboration that helped increase their confidence in Google Cloud.  

Hot off the press: two new papers. For telecommunications organizations in the U.S., our paper Insights into the U.S. Telecommunications Industry addresses key industry trends, regulatory themes, influences, and how companies can use Google Cloud services to tackle their top-of-mind challenges. In support of the European Commission’s digital transformation vision, Google published a paper and blog outlining a set of security recommendations. The recommendations include driving resilience through “open security” and prioritizing strong encryption. 

Using our compliance developments

Many Google Cloud customers already use our trust resources to facilitate internal and external conversations with their key customers, business partners, and regulators. Visit our Compliance Resource Center or continue the conversation with our sales team.

When running Apache Hadoop and Spark, it is important to tune the configs, perform cluster planning, and right-size compute. Thorough benchmarking is required to make sure the utilization and performance are optimized. In Dataproc, you can run Spark jobs in a semi-long-running cluster or ephemeral Cloud Dataproc on Google Compute Engine (DPGCE) cluster or via Dataproc Serverless Spark. Dataproc Serverless for Spark runs a workload on an ephemeral cluster. An ephemeral cluster means the cluster’s lifecycle is tied to the job. A cluster is started, used to run the job, and then destroyed after completion. Ephemeral clusters are easier to configure, since they run a single workload or a few workloads in a sequence. You can leverage Dataproc Workflow Templates to orchestrate this. Ephemeral clusters can be sized to match the job’s requirements. This job-scoped cluster model is effective for batch processing. You can create an ephemeral cluster and configure it to run specific Hive workloads, Apache Pig scripts, Presto queries, etc., and then delete the cluster when the job is completed.  

Ephemeral clusters have some compelling advantages:

They reduce unnecessary storage and service costs from idle clusters and worker machines.

Every set of jobs runs on a job-scoped ephemeral cluster with job-scoped cluster specifications, image version, and operating system. 

Since each job gets a dedicated cluster, the performance of one job does not impact other jobs.

Persistent History Server (PHS)

The challenge with ephemeral clusters and Dataproc Serverless for Spark is that you will lose the application logs when the cluster machines are deleted after the job. Persistent History Server (PHS) enables access to the completed Hadoop and Spark application details for the jobs executed on different ephemeral clusters or serverless Spark. It can list running and completed applications. PHS keeps the history (event logs) of all completed applications and its runtime information in the GCS bucket, and it allows you to review metrics and monitor the application at a later time. PHS is nothing but a standalone cluster. It reads the Spark events from GCS, then parses and presents application details, scheduler stages, and task level details, as well as environment and executor information, in the Spark UI. These metrics are helpful for improving the performance of the application. Both the application event logs and the YARN container logs of the ephemeral clusters are collected in the GCS bucket. These log files are important for engaging Google Cloud Technical Support to troubleshoot and explore. If PHS is not set up, you have to re-run the workload, which adds to support cycle time. If you have set up PHS, you can provide the logs directly to Technical Support.

The following diagram depicts the flow of events logged from ephemeral clusters to the PHS server:

In this blog, we will focus on Dataproc PHS best practices. To set up PHS to access web interfaces of MapReduce and Spark job history files, please refer to Dataproc documentation.

PHS Best Practices

Cluster Planning and Maintenance

It’s common to have a single PHS for a given GCP project. If needed, you can create two or more PHSs pointing to different GCS buckets in a project. This allows you to isolate and monitor specific business applications that run multiple ephemeral jobs and require a dedicated PHS. 

For disaster recovery, you can quickly spin up a new PHS in another region in the event of a zonal or regional failure. 

If you require High Availability (HA), you can spin up two or more PHS instances across zones or regions. All instances can be backed by the dual-regional or multi-regional GCS bucket.

You can run PHS on a single-node Dataproc cluster, as it is not running large-scale parallel processing jobs. For the PHS machine type:

N2 are the most cost-effective and performant machines for Dataproc. We also recommend 500-1000GB pd-standard disks.

For <1000 apps and if there are apps with 50K-100K tasks, we suggest n2-highmem-8.

For <1000 apps and there are apps with 50K-100K tasks, we suggest n2-highmem-8.

For >10000, we suggest n2-highmem16.

We recommend you benchmark with your Spark applications in the testing environment before configuring PHS in production. Once in production, we recommend monitoring your GCE backed PHS instance for memory and CPU utilization and tweaking machine shape as required.

In the event of significant performance degradation within the Spark UI due to a large amount of applications or large jobs generating large event logs, you can recreate the PHS with increased machine size with higher memory. 

As Dataproc releases new sub-minor versions on a bi-weekly cadence or greater, we recommend recreating your PHS instance so it has access to the latest Dataproc binaries and OS security patches. 

As PHS services (e.g. Spark UI, MapReduce History Server) are backwards compatible, it’s suggested to create a Dataproc 2.0+ based PHS cluster for all instances.

Logs Storage 

Configure spark:spark.history.fs.logDirectory to specify where to store event log history written by ephemeral clusters or serverless Spark. You need to create the GCS bucket in advance. 

Event logs are critical for PHS servers. As the event logs are stored in a GCS bucket, it is recommended to use a multi-Region GCS bucket for high availability. Objects inside the multi-region bucket are stored redundantly in at least two separate geographic places separated by at least 100 miles.

Configuration

PHS is stateless and it constructs the Spark UI of the applications by reading the application’s event logs from the GCS bucket. SPARK_DAEMON_MEMORY is the memory to allocate to the history server and has a default of 3840m. If too many users are trying to access the Spark UI and access job application details, or if there are long-running Spark jobs (iterated through several stages with 50K or 100K tasks), the heap size is probably too small. Since there is no way to limit the number of tasks stored on the heap, try increasing the heap size to 8g or 16g until you find a number that works for your scenario.

If you’ve increased the heap size and still see performance issues, you can configure spark.history.retainedApplications and lower the number of retained applications in the PHS.   

Configure mapred:mapreduce.jobhistory.read-only.dir-pattern to access MapReduce job history logs written by ephemeral clusters. 

By default, spark:spark.history.fs.gs.outputstream.type is set to BASIC. The job cluster will send data to GCS after job completion. Set this to FLUSHABLE_COMPOSITE to copy data to GCS at regular intervals while the job is running. 

Configure spark:spark.history.fs.gs.outputstream.sync.min.interval.ms to control the frequency at which the job cluster transfers data to GCS. 

To enable the executor logs in PHS, specify the custom Spark executor log URL for supporting external log service. Configure the following properties:

Lifecycle Management of Logs

Using GCS Object Lifecycle Management, configure a 30d lifecycle policy to periodically clean up the MapReduce job history logs and Spark event logs from the GCS bucket. This will improve the performance of the PHS UI considerably.

Note: Before doing the cleanup, you can  back up the logs to a separate GCS bucket for long-term storage. 

PHS Setup Sample Codes

The following code block creates a Persistent History server with the best practices suggested above.

The following code block creates an ephemeral cluster that logs events to the GCS bucket.

With the Persistent History server, you can monitor and analyze all completed applications. You will also be able to use the logs and metrics to optimize performance and to troubleshoot issues related to strangled tasks, scheduler delays, and out of memory errors. 

Additional Resources

Dataproc PHS Documentation

PHS with Terraform

Spark Monitoring and Instrumentation

Run a Spark Workload

Spark History Web Interface

Monitoring Compute Engine VMs

Today, we published a new Google research report on software supply chain security because we’ve seen a sharp rise in software supply chain attacks across almost every sector —and expect these trends to continue for the foreseeable future. We urge all organizations to act now to improve their software supply chain security.

Among the report’s conclusions, there are two key findings we want to highlight. First, the lessons we’ve learned from various security events call for a more holistic approach to strengthen defenses against software supply chain attacks. Second, we have worked with the security  community to develop and deploy a common Supply-chain Levels for Software Artifacts (SLSA) framework that can mitigate threats across the entire software supply chain ecosystem. These frameworks can help organizations securely build and verify the integrity of software. You can find more information on the report’s conclusions here. 

We know that modern day software supply chains continue to grow deeper, wider, and more complex. That complexity can make it challenging for customers to even know where to begin analyzing their supply chains for security issues. Our research shows that organizations must deal with these same complex issues regardless of which environments they operate in.

At Google Cloud, we’re deeply committed to working with our customers to help ensure that they have the support they need to evaluate their security posture, resiliency, and hygiene. Below, we suggest five steps to protect software across processes and systems, and tap into relevant Google Cloud products and services. These recommendations can enable customers to benefit from Google’s extensive security experience and reduce their need to develop, maintain, and operate complex processes to secure their open source dependencies.

Implementing best practices with Google Cloud

Customers who are interested in improving their software supply chain security can take immediate steps to implement best practices.  

1. Enhance your existing Google Cloud security features with the Google Cloud security foundation guide. The guide can help you weigh important considerations including organizational structure, authentication and authorization, resource hierarchy, networking, logging, and detective controls. You can further engage Mandiant experts to assess your readiness. 

You can also view centralized information about vulnerabilities and possible risks using Google Cloud services like Security Command Center, and get information about your service usage with Recommender, including recommendations that can help you to reduce risk. For example, you can identify IAM principals with excess permissions or unattended Google Cloud projects. You can also find additional resources from the Google Cybersecurity Action Team (GCAT), our premier security advisory team, here.

2. Explore fast software delivery and reliable and secure software with Google Cloud’s DevOps capabilities. You also should review foundational practices for designing, developing, and testing code that apply to most programming languages. 

We strongly recommend you evaluate how you distribute software and the terms of software licenses in all of your dependencies. For more information on Google’s approach to helping organizations address vulnerabilities in open source software, see Appendix B in the research report. 

3. Document the policies for your organization and incorporate validation of policies into your development, build, and deployment processes as you implement best practices. For example, your organization’s policies might include criteria for deployment that you implement with Binary Authorization. GCAT has published additional information on security policies and other cloud security transformation tips for CISOs here. 

You can also explore Minimum Viable Secure Product, a security checklist of controls to establish a baseline security posture for a product. You can use the checklist to establish your minimum security control requirements and to evaluate software by third-party vendors.

Tapping into new Google product and service offerings

At Google Cloud, we continue to focus on delivering new and innovative security capabilities to help customers address the latest security threats. From the attack on SolarWinds to the community response to open source vulnerabilities such as Log4j, we’re seeing a spike in demand from customers on what we can do to help them manage software supply chain risk. We’ve made several recent announcements on that front that can help customers get started with Google Cloud today. 

4. Use Google Cloud’s Software Delivery Shield. It provides a fully managed software supply chain security solution that offers a modular set of capabilities to help equip developers, DevOps, and security teams with the tools they need to build secure cloud applications. Software Delivery Shield spans across a family of Google Cloud services from developer tooling to runtimes including GKE, Cloud Code, Cloud Build, Cloud Deploy, Artifact Registry, and Binary Authorization. To learn more about Software Delivery Shield, check out the solution page, or watch this Google Cloud Next session to get a quick overview of Software Delivery Shield.

5. Enable our Assured Open Source Software (OSS) service, which can help enterprise and public sector open source software users to easily incorporate the same OSS packages that we use at Google into their own developer workflows. Packages curated by the Assured OSS service: 

are regularly scanned, analyzed, and fuzz-tested for vulnerabilities; 

have corresponding enriched metadata incorporating Container/Artifact Analysis data; 

are built with Cloud Build including evidence of verifiable Supply chain Levels for Software Artifacts (SLSA)-compliance;

are verifiably signed by Google; 

and are distributed from an Artifact Registry that is secured and protected by Google. 

If you are interested in learning more about software supply chain security in general, please contact usor reach out to your sales representative to schedule a software supply chain security workshop.

Did you know that most Google employees rely on virtual desktops to get their work done? This represents a paradigm shift in client computing at Google, and was especially critical during the pandemic and the remote work revolution. We’re excited to continue enabling our employees to be productive, anywhere! This post covers the history of virtual desktops and details the numerous benefits Google has seen from their implementation. 

Background

In 2018, Google began the development of virtual desktops in the cloud. A whitepaper was published detailing how virtual desktops were created with Google Cloud, running on Google Compute Engine, as an alternative to physical workstations. Further research had shown that it was feasible to move our physical workstation fleet to these virtual desktops in the cloud. The research began with user experience analysis – looking into how employee satisfaction of cloud workstations compared with physical desktops. Researchers found that user satisfaction of cloud desktops was higher than that of their physical desktop counterparts! This was a monumental moment for cloud-based client computing at Google, and this discovery led to additional analyses of Compute Engine to understand if it could become our preferred (virtual) workstation platform of the future.

Today, Google’s internal use of virtual desktops has increased dramatically. Employees all over the globe use a mix of virtual Linux and Windows desktops on Compute Engine to complete their work. Whether an employee is writing code, accessing production systems, troubleshooting issues, or driving productivity initiatives, virtual desktops are providing them with the compute they need to get their work done. Access to virtual desktops is simple: some employees access their virtual desktop instances via Secure Shell (SSH), while others use Chrome Remote Desktop — a graphical access tool. 

In addition to simplicity and accessibility, Google has realized a number of benefits from virtual desktops. We’ve seen an enhanced security posture, a boost to our sustainability initiatives, and a reduction in maintenance effort associated with our IT infrastructure. All these improvements were achieved while improving the user experience compared to our physical workstation fleet.

Analyzing Cloud vs Physical Desktops

Let’s look deeper into the analysis Google performed to compare cloud virtual desktops and physical desktops. Researchers compared cloud and physical desktops on five core pillars: user experience, performance, sustainability, security, and efficiency.

User Experience

Before the transition to virtual desktops got underway, user experience researchers wanted to know more about how they would affect employee happiness. They discovered that employees embraced the benefits that virtual desktops offered. This included freeing up valuable desk space to provide an always-on, always available compute experience, accessible from anywhere in the world, and reduced maintenance overhead compared to physical desktops. 

Performance

From a performance perspective, cloud desktops are simply better than physical desktops. For example, running on Compute Engine makes it easy to spin-up on-demand virtual instances with predictable compute and performance – a task that is significantly more difficult with a physical workstation vendor. Virtual desktops rely on a mix of Virtual Machine (VM) families that Google developed based on the performance needs of our users. These include Google Compute EngineE2 high-efficiency instances, which employees might use for day-to-day tasks, to higher-performance N2/N2D instances, which employees might use for more demanding machine learning jobs. Compute Engine offers a VM shape for practically any computing workflow. Additionally, employees no longer have to worry about machine upgrades (to increase performance, for example) because our entire fleet of virtual desktops can be upgraded to new shapes (with more CPU and RAM) with a single config change and a simple reboot — all within a matter of minutes. Plus, Compute Engine continues to add features and new machine types, which means our capabilities only continue to grow in this space.

Sustainability

Google cares deeply about sustainability and has been carbon neutral since 2007. Moving from physical desktops to virtual desktops on Compute Engine brings us closer to Google sustainability goals of a net-neutral desktop computing fleet. Our internal facilities team has praised virtual desktops as a win for future workspace planning, because a reduction in physical workstations could also mean a reduction in first-time construction costs of new buildings, significant (up to 30%) campus energy reductions, and even further reductions in costs associated with HVAC needs and circuit size needs at our campuses. Lastly, a reduction in physical workstations also contributes to a reduction in physical e-waste and a reduction in the carbon associated with transporting workstations from their factory of origin to office locations. At Google’s scale, these changes lead to an immense win from a sustainability standpoint. 

Security

By their very nature, virtual desktops mitigate the ability for a bad actor to exfiltrate data or otherwise compromise physical desktop hardware since there is no desktop hardware to compromise in the first place. This means attacks such as USB attacks, evil maid attacks, and similar techniques for subverting security that require direct hardware access become worries of the past. Additionally, the transition to cloud-based virtual desktops also brings with it an enhanced security posture through the use of Google Cloud’s myriad security features including Confidential Computing, vTPMs, and more. 

Efficiency

In the past, it was not uncommon for employees to spend days waiting for IT to deliver new machines or fix physical workstations. Today, cloud-based desktops can be created instantaneously on-demand and resized on-demand. They are always accessible, and virtually immune from maintenance-related issues. IT no longer has to deal with concerns like warranty claims, break-fix issues, or recycling. This time savings enables IT to focus on higher priority initiatives all while reducing their workload. With an enterprise the size of Google, these efficiency wins added up quickly. 

Considerations to Keep in Mind

Although Google has seen significant benefits with virtual desktops, there are some considerations to keep in mind before deciding if they are right for your enterprise. First, it’s important to recognize that migrating to a virtual fleet requires a consistently reliable and performant client internet connection. For remote/global employees, it’s important they’re located geographically near a Google Cloud Region (to minimize latency). Additionally, there are cases where physical workstations are still considered vital. These cases include users who need USB and other direct I/O access for testing/debugging hardware and users who have ultra low-latency graphics/video editing or CAD simulation needs. Finally, to ensure interoperability between these virtual desktops and the rest of our computing fleet, we did have to perform some additional engineering tasks to integrate our asset management and other IT systems with the virtual desktops. Whether your enterprise needs such features and integration should be carefully analyzed before considering a solution such as this. However, should you ultimately conclude that cloud-based desktops are the solution for your enterprise, we’re confident you’ll realize many of the benefits we have!

Tying It All Together

Although moving Google employees to virtual desktops in the clouds was a significant engineering undertaking, the benefits have been just as significant.  Making this switch has boosted employee productivity and satisfaction, enhanced security, increased efficiency, and provided noticeable improvements in performance and user experience. In short, cloud-based desktops are helping us transform how Googlers get their work done. During the pandemic, we saw the benefits of virtual desktops in a critical time. Employees had access to their virtual desktop from anywhere in the world, which kept our workforce safer and reduced transmission vectors for COVID-19. We’re excited for a future where more and more of our employees are computing in the cloud as we continue to embrace the work-from-anywhere model and as we continue to add new features and enhanced capabilities to Compute Engine!