Skip to content

Slurm Cluster Blueprint for GCP

General Info

This page provides an example Cluster Toolkit blueprint for use with ParaTools Pro for E4S™. Once you have subscribed to ParaTools Pro for E4S™ on the GCP Marketplace, we recommend working through the "Deploy an HPC cluster with Slurm" quickstart from the Cluster Toolkit project if you are new to GCP or to the Cluster Toolkit. The blueprint below can be copied with small modifications and used either for the tutorial or in production.

Areas of the blueprint that require your attention, and that may need to be changed, are highlighted and have expandable annotations offering further guidance.

ParaTools Pro for E4S™ Slurm Cluster Blueprint Example

e4s-25.11-cluster-slurm-gcp-v6.yaml
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

---

blueprint_name: ppro-e4s-25-11-cluster

vars:
  project_id: paratools-pro #(1)!
  deployment_name: ppro-e4s-25-11-cluster
  region: us-central1
  zone: us-central1-a
  disk_size: 250
  image_family: paratools-gcluster-e4s-2511-nvidia89-x86-64 #(2)!
  instance_image_custom: true

# Documentation for each of the modules used below can be found at
# https://github.com/GoogleCloudPlatform/cluster-toolkit/blob/main/modules/README.md

deployment_groups:
- group: primary
  modules:
  # Source is an embedded module, denoted by "modules/*" without ./, ../, /
  # as a prefix. To refer to a local module, prefix with ./, ../ or /
  - id: network
    source: modules/network/vpc #(3)!

  - id: homefs
    source: modules/file-system/filestore
    use: [network]
    settings:
      local_mount: /home

  - id: debug_nodeset
    source: community/modules/compute/schedmd-slurm-gcp-v6-nodeset
    use: [network]
    settings:
      node_count_dynamic_max: 2 #(4)!
      machine_type: n1-standard-16
      guest_accelerator: #(5)!
      - type: nvidia-tesla-t4
        count: 1
      enable_placement: false # the default is: true
      instance_image_custom: true
      disk_size_gb: $(vars.disk_size)
      instance_image:
        family: $(vars.image_family)
        project: $(vars.project_id)

  - id: debug_partition
    source: community/modules/compute/schedmd-slurm-gcp-v6-partition
    use:
    - debug_nodeset
    settings:
      partition_name: debug
      exclusive: false # allows nodes to stay up after jobs are done
      is_default: true

  - id: compute_nodeset
    source: community/modules/compute/schedmd-slurm-gcp-v6-nodeset
    use: [network]
    settings:
      node_count_dynamic_max: 2
      machine_type: c3-standard-88
      bandwidth_tier: tier_1_enabled #(6)!
      instance_image_custom: true
      disk_size_gb: $(vars.disk_size)
      instance_image:
        family: $(vars.image_family)
        project: $(vars.project_id)

  - id: compute_partition
    source: community/modules/compute/schedmd-slurm-gcp-v6-partition
    use:
    - compute_nodeset
    settings:
      partition_name: compute

  - id: h3_nodeset #(7)!
    source: community/modules/compute/schedmd-slurm-gcp-v6-nodeset
    use: [network]
    settings:
      node_count_dynamic_max: 2
      machine_type: h3-standard-88
      # H3 does not support pd-ssd and pd-standard
      # https://cloud.google.com/compute/docs/compute-optimized-machines#h3_disks
      disk_type: pd-balanced
      bandwidth_tier: gvnic_enabled
      instance_image_custom: true
      disk_size_gb: $(vars.disk_size)
      instance_image:
        family: $(vars.image_family)
        project: $(vars.project_id)

  - id: h3_partition
    source: community/modules/compute/schedmd-slurm-gcp-v6-partition
    use:
    - h3_nodeset
    settings:
      partition_name: h3

  - id: slurm_login
    source: community/modules/scheduler/schedmd-slurm-gcp-v6-login
    use: [network]
    settings:
      machine_type: n1-standard-16
      enable_login_public_ips: true
      instance_image_custom: true
      disk_size_gb: $(vars.disk_size)
      instance_image:
        family: $(vars.image_family)
        project: $(vars.project_id)

  - id: slurm_controller
    source: community/modules/scheduler/schedmd-slurm-gcp-v6-controller
    use:
    - network
    - debug_partition
    - compute_partition
    - h3_partition
    - homefs
    - slurm_login
    settings:
      enable_controller_public_ips: true
      disk_size_gb: $(vars.disk_size)
      instance_image:
        family: $(vars.image_family)
        project: $(vars.project_id)

  1. Use your own GCP project ID

    Set this to your own GCP project ID, or comment this line and invoke gcluster with the --vars project_id="${PROJECT_ID}" flag instead. The value shown here is the ParaTools development project and will not work for you.

  2. Image family

    This must match the image family of the ParaTools Pro for E4S™ marketplace image you have subscribed to. The current GCluster (x86-64) image family is paratools-gcluster-e4s-2511-nvidia89-x86-64.

  3. Default VPC firewall behavior

    The modules/network/vpc module with no settings: block creates a VPC with two firewall rules by default: one allowing SSH from Identity-Aware Proxy (IAP) (range 35.235.240.0/20) and one allowing all intra-VPC traffic. The SSH button in the GCP Console for the login node uses IAP, so it works with this default. If you want to SSH directly to the login node's public IP from your workstation, you must add a firewall rule that allows your workstation's IP -- see Allowing direct SSH from your workstation.

  4. Node count and instance type

    The machine_type and node_count_dynamic_max set the instance type and the maximum number of nodes Slurm can spin up dynamically in this nodeset. Tune these to match your usage and quotas (vCPUs = cores-per-node x node_count_dynamic_max). This same setting also appears in the compute_nodeset (line 74) and h3_nodeset (line 94) blocks. For compute-intensive, tightly coupled jobs, C3 or H3 instances perform well.

  5. GPU on the debug nodeset

    The debug_nodeset attaches one NVIDIA Tesla T4 GPU per node by default, providing a low-cost path to test CUDA, NeMo, or PyTorch GPU workloads before scaling up. Set count: 0 and remove the guest_accelerator block, or change to a CPU-only instance type, if you do not need a GPU on the debug partition.

  6. bandwidth_tier: gVNIC vs. Tier 1

    The bandwidth_tier setting controls the network adapter and per-VM egress bandwidth ceiling. Valid values are platform_default, virtio_enabled, gvnic_enabled (the gVNIC adapter without Tier 1), and tier_1_enabled (gVNIC adapter plus per-VM Tier 1 high-bandwidth networking). The compute_nodeset uses tier_1_enabled because c3-standard-88 is a C3 shape with 88 vCPUs, which clears the Tier 1 minimum (44 vCPUs for C3) and unlocks the full bandwidth ceiling for tightly coupled MPI and collective-heavy workloads. The h3_nodeset (line 99) uses gvnic_enabled because the H3 family is not in the Tier 1 supported list (despite its 88 vCPUs); H3 is limited to gVNIC speeds. If you change compute_nodeset's machine_type to a shape that does not support Tier 1 (for example any N1, N2, E2, or H3 shape, or a supported family below its vCPU minimum), drop tier_1_enabled back to gvnic_enabled to avoid a deployment error.

  7. Optional H3 partition

    This example includes an additional Slurm partition containing H3 nodes. Access to H3 instances may require a quota-increase request. You do not need multiple Slurm partitions, so you may remove the h3_nodeset and h3_partition modules (and the - h3_partition reference in slurm_controller.use) if you do not have H3 access.

Allowing Direct SSH from Your Workstation

The default firewall rules created by modules/network/vpc permit SSH from Identity-Aware Proxy (IAP) only. The SSH button in the GCP Console (which uses IAP) works without any extra configuration. If you want to SSH directly from your workstation to the login node's public IP -- for example, to use scp for large file transfers, or because you prefer a local terminal over the browser-based IAP SSH session -- you must allow your workstation's IP address through the firewall.

There are two equivalent options:

Option 1: Add the Rule to the Blueprint

Edit the network module in your blueprint and add a settings: block listing the firewall rule. Replace 203.0.113.42/32 with your workstation's public IP (find it with curl -s ifconfig.me), or replace 203.0.113.0/24 with a CIDR block covering your home or office network:

  - id: network
    source: modules/network/vpc
    settings:
      firewall_rules:
        - name: ssh-from-workstation
          direction: INGRESS
          ranges: [203.0.113.42/32]   # single IP -- replace with your workstation's IP
          # ranges: [203.0.113.0/24] # or a CIDR block covering your network
          allow:
            - protocol: tcp
              ports: [22]

Then re-run gcluster create -w ... and gcluster deploy ... to apply the change.

Option 2: Add the Rule Out of Band

After the cluster is deployed, add the firewall rule with gcloud directly:

# Replace YOUR_IP with your workstation's public IP, or a CIDR covering your network.
# Replace VPC_NAME with the name of the VPC created by your deployment
# (typically "${deployment_name}-net0", e.g., "ppro-e4s-25-11-cluster-net0").
gcloud compute firewall-rules create ssh-from-workstation \
    --network=VPC_NAME \
    --direction=INGRESS \
    --action=ALLOW \
    --rules=tcp:22 \
    --source-ranges=YOUR_IP/32

Do not use 0.0.0.0/0

Opening TCP port 22 to the entire internet is a serious security risk. Always restrict --source-ranges (or ranges:) to a single IP (/32) or a small CIDR block under your control. Prefer Console (IAP) SSH whenever possible.