Add GCP Compute Engine backend

docs/source/compute_backends.rst

@@ -44,3 +44,4 @@ Compute Backends
    compute_config/ibm_vpc.md
    compute_config/aws_ec2.md
    compute_config/azure_vms.md
+   compute_config/gcp_compute_engie.md

docs/source/compute_config/gcp_compute_engie.md

@@ -0,0 +1,122 @@
+# Google Compute Engine (GCE)
+
+Lithops includes a standalone backend named `gcp_compute_engie` for running jobs on Google Compute Engine virtual machines.
+
+This backend supports:
+
+- **consume mode**: use an existing VM instance.
+- **create mode**: create the network resources and master/worker VMs on demand.
+- **reuse mode**: keep and reuse workers between executions, creating only the missing delta.
+
+> **Note**
+> The backend key is intentionally `gcp_compute_engie` to match the current Lithops backend module name.
+
+## Installation
+
+Install Google Cloud dependencies:
+
+```bash
+python3 -m pip install lithops[gcp]
+```
+
+## Required IAM/API setup
+
+Use a service account with permissions to manage Compute Engine resources (instances, networks, subnetworks, firewalls) and enable:
+
+- Compute Engine API (`compute.googleapis.com`)
+
+Optional command:
+
+```bash
+gcloud services enable compute.googleapis.com --project <PROJECT_ID>
+```
+
+## Create and reuse modes
+
+In `create` mode, Lithops automatically provisions VPC/network resources plus worker VMs during execution and dismantles workers afterward.  
+In `reuse` mode, Lithops reuses existing workers and only creates additional workers when needed.
+
+### Configuration
+
+```yaml
+lithops:
+  backend: gcp_compute_engie
+  mode: standalone
+
+gcp:
+  credentials_path: <FULL_PATH_TO_SERVICE_ACCOUNT_JSON>
+
+gcp_compute_engie:
+  project_name: <GCP_PROJECT_ID>
+  zone: <ZONE>            # e.g. us-east1-b
+  region: <REGION>        # optional, derived from zone if omitted
+  exec_mode: reuse        # create | reuse | consume
+  worker_instance_type: e2-standard-2
+  master_instance_type: e2-small
+```
+
+### Summary of configuration keys (create/reuse)
+
+|Group|Key|Default|Mandatory|Additional info|
+|---|---|---|---|---|
+|gcp_compute_engie|project_name||yes|GCP project ID |
+|gcp_compute_engie|zone||yes|Compute Engine zone, for example `us-east1-b` |
+|gcp_compute_engie|region|derived from zone|no|Region used for subnet creation |
+|gcp_compute_engie|credentials_path||no|Service account JSON path. If omitted, ADC is used |
+|gcp_compute_engie|master_instance_type|e2-small|no|Master VM machine type |
+|gcp_compute_engie|worker_instance_type|e2-standard-2|no|Worker VM machine type |
+|gcp_compute_engie|source_image|ubuntu-2204-lts family|no|Boot image reference |
+|gcp_compute_engie|boot_disk_size|50|no|Boot disk size (GB) |
+|gcp_compute_engie|boot_disk_type|pd-standard|no|Boot disk type |
+|gcp_compute_engie|network_cidr|10.0.0.0/16|no|CIDR for created network |
+|gcp_compute_engie|subnet_cidr|10.0.0.0/24|no|CIDR for created subnet |
+|gcp_compute_engie|request_spot_instances|False|no|Use Spot VMs for workers |
+|gcp_compute_engie|max_workers|100|no|Max number of workers per `FunctionExecutor()` |
+|gcp_compute_engie|worker_processes|AUTO|no|Worker process count |
+|gcp_compute_engie|exec_mode|reuse|no|One of `consume`, `create`, `reuse` |
+
+## Consume mode
+
+In consume mode, Lithops uses an existing VM and does not provision network/worker resources.
+
+### Configuration
+
+```yaml
+lithops:
+  backend: gcp_compute_engie
+  mode: standalone
+
+gcp:
+  credentials_path: <FULL_PATH_TO_SERVICE_ACCOUNT_JSON>
+
+gcp_compute_engie:
+  exec_mode: consume
+  project_name: <GCP_PROJECT_ID>
+  zone: <ZONE>
+  instance_name: <EXISTING_VM_NAME>
+  ssh_username: ubuntu
+  ssh_key_filename: ~/.ssh/id_rsa
+```
+
+### Summary of configuration keys (consume)
+
+|Group|Key|Default|Mandatory|Additional info|
+|---|---|---|---|---|
+|gcp_compute_engie|project_name||yes|GCP project ID |
+|gcp_compute_engie|zone||yes|Compute Engine zone |
+|gcp_compute_engie|instance_name||yes|Existing VM instance name |
+|gcp_compute_engie|ssh_username|ubuntu|no|SSH user for the VM |
+|gcp_compute_engie|ssh_key_filename|~/.ssh/id_rsa|no|Path to SSH private key |
+|gcp_compute_engie|worker_processes|AUTO|no|Worker process count |
+
+## Test Lithops
+
+```bash
+lithops hello -b gcp_compute_engie -s gcp_storage
+```
+
+## Viewing execution logs
+
+```bash
+lithops logs poll
+```

docs/source/configuration.rst

@@ -37,6 +37,7 @@ Choose your compute and storage engines from the table below:
 || `IBM Virtual Private Cloud <compute_config/ibm_vpc.html>`_        ||                                                                   |
 || `AWS Elastic Compute Cloud (EC2) <compute_config/aws_ec2.html>`_  ||                                                                   |
 || `Azure Virtual Machines <compute_config/azure_vms.html>`_         ||                                                                   |
+|| `Google Compute Engine <compute_config/gcp_compute_engie.html>`_  ||                                                                   |
 +--------------------------------------------------------------------+--------------------------------------------------------------------+
 
 Configuration File

docs/source/execution_modes.rst

@@ -83,4 +83,4 @@ underlying infrastructure.
 
     fexec = lithops.StandaloneExecutor()
 
-- Available backends: `IBM Virtual Private Cloud <compute_config/ibm_vpc.md>`_, `AWS Elastic Compute Cloud (EC2) <compute_config/aws_ec2.md>`_, `Azure Virtual Machines <compute_config/azure_vms.md>`_, `Virtual Machine <compute_config/vm.md>`_
+- Available backends: `IBM Virtual Private Cloud <compute_config/ibm_vpc.md>`_, `AWS Elastic Compute Cloud (EC2) <compute_config/aws_ec2.md>`_, `Azure Virtual Machines <compute_config/azure_vms.md>`_, `Google Compute Engine <compute_config/gcp_compute_engie.md>`_, `Virtual Machine <compute_config/vm.md>`_

docs/source/supported_clouds.rst

@@ -52,6 +52,8 @@ Currently, Lithops for Google Cloud Platform supports these backends:
      - `Google Cloud Storage <https://cloud.google.com/storage/docs>`_
    * - `Google Cloud Run <https://cloud.google.com/run/docs>`_
      -
+   * - `Google Compute Engine (GCE) <https://cloud.google.com/compute/docs>`_
+     -
 
 Microsoft Azure
 ---------------

lithops/standalone/backends/gcp_compute_engie/__init__.py

@@ -0,0 +1,3 @@
+from .gcp_compute_engie import GCPComputeEngieBackend as StandaloneBackend
+
+__all__ = ['StandaloneBackend']

lithops/standalone/backends/gcp_compute_engie/config.py

@@ -0,0 +1,90 @@
+#
+# Copyright Cloudlab URV 2021
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import copy
+import os
+import uuid
+
+from lithops.constants import SA_DEFAULT_CONFIG_KEYS
+
+
+DEFAULT_CONFIG_KEYS = {
+    'master_instance_type': 'e2-small',
+    'worker_instance_type': 'e2-standard-2',
+    'boot_disk_size': 50,
+    'boot_disk_type': 'pd-standard',
+    'network_cidr': '10.0.0.0/16',
+    'subnet_cidr': '10.0.0.0/24',
+    'source_image': 'projects/ubuntu-os-cloud/global/images/family/ubuntu-2204-lts',
+    'ssh_username': 'ubuntu',
+    'ssh_password': str(uuid.uuid4()),
+    'ssh_key_filename': '~/.ssh/id_rsa',
+    'delete_on_dismantle': False,
+    'max_workers': 100,
+    'request_spot_instances': False,
+    'worker_processes': 'AUTO'
+}
+
+MANDATORY_PARAMETERS_1 = ('project_name', 'zone', 'instance_name')
+MANDATORY_PARAMETERS_2 = ('project_name', 'zone')
+
+
+def load_config(config_data):
+    if 'gcp_compute_engie' not in config_data or not config_data['gcp_compute_engie']:
+        raise Exception("'gcp_compute_engie' section is mandatory in the configuration")
+
+    if 'gcp' not in config_data:
+        config_data['gcp'] = {}
+
+    temp = copy.deepcopy(config_data['gcp_compute_engie'])
+    config_data['gcp_compute_engie'].update(config_data['gcp'])
+    config_data['gcp_compute_engie'].update(temp)
+
+    if 'credentials_path' in config_data['gcp_compute_engie']:
+        config_data['gcp_compute_engie']['credentials_path'] = os.path.expanduser(
+            config_data['gcp_compute_engie']['credentials_path']
+        )
+
+    for key in DEFAULT_CONFIG_KEYS:
+        if key not in config_data['gcp_compute_engie']:
+            config_data['gcp_compute_engie'][key] = DEFAULT_CONFIG_KEYS[key]
+
+    if 'standalone' not in config_data or config_data['standalone'] is None:
+        config_data['standalone'] = {}
+
+    for key in SA_DEFAULT_CONFIG_KEYS:
+        if key in config_data['gcp_compute_engie']:
+            config_data['standalone'][key] = config_data['gcp_compute_engie'].pop(key)
+        elif key not in config_data['standalone']:
+            config_data['standalone'][key] = SA_DEFAULT_CONFIG_KEYS[key]
+
+    if config_data['standalone']['exec_mode'] == 'consume':
+        params_to_check = MANDATORY_PARAMETERS_1
+        config_data['gcp_compute_engie']['max_workers'] = 1
+    else:
+        params_to_check = MANDATORY_PARAMETERS_2
+
+    for param in params_to_check:
+        if param not in config_data['gcp_compute_engie']:
+            msg = f"'{param}' is mandatory in the 'gcp_compute_engie' section of the configuration"
+            raise Exception(msg)
+
+    if 'region' not in config_data['gcp_compute_engie']:
+        zone = config_data['gcp_compute_engie']['zone']
+        config_data['gcp_compute_engie']['region'] = '-'.join(zone.split('-')[:-1])
+
+    if 'region' not in config_data['gcp'] and 'region' in config_data['gcp_compute_engie']:
+        config_data['gcp']['region'] = config_data['gcp_compute_engie']['region']