[wip] remove usage

.github/workflows/README.md

@@ -236,6 +236,7 @@ PreCommit Jobs run in a schedule and also get triggered in a PR if relevant sour
 | [ PreCommit GHA ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_GHA.yml) | N/A |`Run GHA PreCommit`| [![.github/workflows/beam_PreCommit_GHA.yml](https://github.com/apache/beam/actions/workflows/beam_PreCommit_GHA.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_GHA.yml?query=event%3Aschedule) |
 | [ PreCommit Go ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Go.yml) | N/A |`Run Go PreCommit`| [![.github/workflows/beam_PreCommit_Go.yml](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Go.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Go.yml?query=event%3Aschedule) |
 | [ PreCommit GoPortable ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_GoPortable.yml) | N/A |`Run GoPortable PreCommit`| [![.github/workflows/beam_PreCommit_GoPortable.yml](https://github.com/apache/beam/actions/workflows/beam_PreCommit_GoPortable.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_GoPortable.yml?query=event%3Aschedule) |
+| [ PreCommit Gsutil Check ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Gsutil_Check.yml) | N/A |`Run Gsutil Check PreCommit`| [![.github/workflows/beam_PreCommit_Gsutil_Check.yml](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Gsutil_Check.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Gsutil_Check.yml?query=event%3Aschedule) |
 | [ PreCommit Java ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java.yml) | N/A |`Run Java PreCommit`| [![.github/workflows/beam_PreCommit_Java.yml](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java.yml?query=event%3Aschedule) |
 | [ PreCommit Java Dataflow Non-portable Worker ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Dataflow.yml) | N/A |`Run Java Dataflow Non-portable Worker PreCommit`| [![.github/workflows/beam_PreCommit_Java_Dataflow.yml](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Dataflow.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Dataflow.yml?query=event%3Aschedule) |
 | [ PreCommit Java Amazon Web Services2 IO Direct ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Amazon-Web-Services2_IO_Direct.yml) | N/A |`Run Java_Amazon-Web-Services2_IO_Direct PreCommit`| [![.github/workflows/beam_PreCommit_Java_Amazon-Web-Services2_IO_Direct.yml](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Amazon-Web-Services2_IO_Direct.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Amazon-Web-Services2_IO_Direct.yml?query=event%3Aschedule) |

.github/workflows/beam_PreCommit_Gsutil_Check.yml

@@ -0,0 +1,102 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This PreCommit Gsutil Check is to ensure no new code enters the repo, since
+# `gsutil` is being replaced by `gcloud storage` officially no later than 
+# March 2027 (Reference: https://docs.cloud.google.com/storage/docs/gsutil).
+# Beginning in April 2027, this workflow will start erroring out and will need
+# to be deleted if `gsutil` was officially deprecated in March 2027 still.
+
+name: PreCommit Gsutil Check
+
+on:
+  push:
+    tags: ['v*']
+    branches: ['master', 'release-*']
+  pull_request_target:
+    branches: ['master', 'release-*']
+  issue_comment:
+    types: [created]
+  schedule:
+    - cron: '15 3/6 * * *'
+  workflow_dispatch:
+
+# Setting explicit permissions for the action to avoid the default permissions which are `write-all` in case of pull_request_target event
+permissions:
+  actions: write
+  pull-requests: read
+  checks: read
+  contents: read
+  deployments: read
+  id-token: none
+  issues: read
+  discussions: read
+  packages: read
+  pages: read
+  repository-projects: read
+  security-events: read
+  statuses: read
+
+# This allows a subsequently queued workflow run to interrupt previous runs
+concurrency:
+  group: '${{ github.workflow }} @ ${{ github.event.pull_request.number || github.event.pull_request.head.label || github.sha || github.head_ref || github.ref }}-${{ github.event.schedule || github.event.comment.id || github.event.sender.login }}'
+  cancel-in-progress: true
+
+env:
+  DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
+  GRADLE_ENTERPRISE_CACHE_USERNAME: ${{ secrets.GE_CACHE_USERNAME }}
+  GRADLE_ENTERPRISE_CACHE_PASSWORD: ${{ secrets.GE_CACHE_PASSWORD }}
+
+jobs:
+  beam_PreCommit_Gsutil_Check:
+    name: ${{ matrix.job_name }} (${{ matrix.job_phrase }})
+    runs-on: [self-hosted, ubuntu-24.04, small]
+    strategy:
+      matrix:
+        job_name: [beam_PreCommit_Gsutil_Check]
+        job_phrase: [Run Gsutil Check PreCommit]
+    timeout-minutes: 10
+    if: |
+      github.event_name == 'push' ||
+      github.event_name == 'pull_request_target' ||
+      (github.event_name == 'schedule' && github.repository == 'apache/beam') ||
+      github.event_name == 'workflow_dispatch' ||
+      github.event.comment.body == 'Run Gsutil Check PreCommit'
+    steps:
+      - uses: actions/checkout@v7
+        with:
+          persist-credentials: false
+      - name: Setup repository
+        uses: ./.github/actions/setup-action
+        with:
+          comment_phrase: ${{ matrix.job_phrase }}
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          github_job: ${{ matrix.job_name }} (${{ matrix.job_phrase }})
+      - name: Check for gsutil references
+        run: |
+          echo "Checking codebase for gsutil..."
+          # Search for 'gsutil', excluding this workflow file itself to avoid false positives.
+          if git grep -n "gsutil" -- ':!.github/workflows/beam_PreCommit_Gsutil_Check.yml'; then
+            echo "ERROR: Found references to gsutil in the codebase. Please use gcloud storage instead."
+            exit 1
+          elif [ "$(date +%Y%m)" -ge 202704 ]; then
+            echo "ERROR: Current date is April 2027 or later."
+            echo "Please verify gsutil deprecation date is still March 2027 (Reference: https://docs.cloud.google.com/storage/docs/gsutil)."
+            echo "If so, then delete this workflow."
+            exit 1
+          else
+            echo "SUCCESS: No references to gsutil found."
+          fi
+        shell: bash

.github/workflows/build_wheels.yml

@@ -201,7 +201,7 @@ jobs:
     if: needs.check_env_variables.outputs.gcp-variables-set == 'true' && github.event_name != 'pull_request'
     steps:
       - name: Remove existing files on GCS bucket
-        run: gsutil rm -r ${GCP_PATH} || true
+        run: gcloud storage rm -r ${GCP_PATH} || true
 
   upload_source_to_gcs:
     name: Upload python source distribution to GCS bucket
@@ -217,7 +217,7 @@ jobs:
           name: source_zip
           path: source/
       - name: Copy sources to GCS bucket
-        run: gsutil cp -r -a public-read source/* ${GCP_PATH}
+        run: gcloud storage cp -r --predefined-acl=publicRead source/* ${GCP_PATH}
 
   build_wheels:
     name: Build python ${{matrix.py_version}} wheels on ${{matrix.os_python.arch}} for ${{ matrix.os_python.os }}
@@ -330,7 +330,7 @@ jobs:
         merge-multiple: true
         path: wheelhouse/
     - name: Copy wheels to GCS bucket
-      run: gsutil cp -r -a public-read wheelhouse/* ${GCP_PATH}
+      run: gcloud storage cp -r --predefined-acl=publicRead wheelhouse/* ${GCP_PATH}
     - name: Create github action information file on GCS bucket
       run: |
         cat > github_action_info <<EOF
@@ -348,9 +348,9 @@ jobs:
         GITHUB_BASE_REF=$GITHUB_BASE_REF
         EOF
         echo $(cat github_action_info)
-        gsutil cp -a public-read github_action_info ${GCP_PATH}
+        gcloud storage cp --predefined-acl=publicRead github_action_info ${GCP_PATH}
     - name: Upload GitHub event file to GCS bucket
-      run: gsutil cp -a public-read ${GITHUB_EVENT_PATH} ${GCP_PATH}
+      run: gcloud storage cp --predefined-acl=publicRead ${GITHUB_EVENT_PATH} ${GCP_PATH}
 
   list_files_on_gcs:
     name: List files on Google Cloud Storage Bucket
@@ -361,7 +361,7 @@ jobs:
     if: needs.check_env_variables.outputs.gcp-variables-set == 'true' && github.event_name != 'pull_request'
     steps:
     - name: List file on Google Cloud Storage Bucket
-      run: gsutil ls "${GCP_PATH}*"
+      run: gcloud storage ls "${GCP_PATH}*"
 
   branch_repo_nightly:
     permissions:

.github/workflows/run_rc_validation_go_wordcount.yml

@@ -74,7 +74,7 @@ jobs:
       # --- Prism Steps ---
       - name: Download Input File from GCS (Prism)
         working-directory: ./${{ steps.setup_go.outputs.work_dir }}
-        run: gsutil cp ${{ env.GCS_INPUT_PATH }} ./kinglear.txt
+        run: gcloud storage cp ${{ env.GCS_INPUT_PATH }} ./kinglear.txt
 
       - name: Run Go WordCount with PrismRunner
         working-directory: ./${{ steps.setup_go.outputs.work_dir }}
@@ -118,17 +118,17 @@ jobs:
             --environment_config=apache/beam_go_sdk:${{ env.CONTAINER_TAG }}
 
       - name: Check Dataflow Output in GCS
-        working-directory: ./${{ steps.setup_go.outputs.work_dir }} # Added working directory for consistency, though not strictly needed for gsutil
+        working-directory: ./${{ steps.setup_go.outputs.work_dir }} # Added working directory for consistency, though not strictly needed for gcloud storage
         env:
           # Re-define the output path pattern for checking
           GCS_OUTPUT_PATH_PATTERN: ${{ env.GCS_OUTPUT_PREFIX }}/${{ env.RC_TAG }}/dataflow/output*
         run: |
           echo "Checking for Dataflow output files in GCS at: $GCS_OUTPUT_PATH_PATTERN"
-          # Use gsutil stat. The -q flag suppresses errors for non-existent files,
-          # allowing us to check the exit code. Exit code 0 means found, 1 means not found.
-          if gsutil -q stat $GCS_OUTPUT_PATH_PATTERN; then
+          # Use gcloud storage ls. Redirect stdout/stderr to suppress output.
+          # Exit code 0 means found, non-zero means not found.
+          if gcloud storage ls $GCS_OUTPUT_PATH_PATTERN > /dev/null 2>&1; then
             echo "Output files found in GCS."
-            FILE_COUNT=$(gsutil ls $GCS_OUTPUT_PATH_PATTERN | wc -l)
+            FILE_COUNT=$(gcloud storage ls $GCS_OUTPUT_PATH_PATTERN | wc -l)
             if [ "$FILE_COUNT" -gt 0 ]; then echo "Found $FILE_COUNT output file(s)."; else echo "Error: Output path exists but contains no files."; exit 1; fi
           else
             echo "Error: Output files not found in GCS at $GCS_OUTPUT_PATH_PATTERN"

.github/workflows/run_rc_validation_python_mobile_gaming.yml

@@ -174,7 +174,7 @@ jobs:
       - name: Create GCS Bucket (if needed - reusing input bucket)
         run: |
           echo "Ensuring GCS Bucket exists: ${{ env.GCS_BUCKET }} in project ${{ env.GCP_PROJECT_ID }}"
-          gsutil mb -p ${{ env.GCP_PROJECT_ID }} ${{ env.GCS_BUCKET }} || echo "Bucket ${{ env.GCS_BUCKET }} likely already exists."
+          gcloud storage buckets create ${{ env.GCS_BUCKET }} --project=${{ env.GCP_PROJECT_ID }} || echo "Bucket ${{ env.GCS_BUCKET }} likely already exists."
         shell: bash
 
       - name: Create PubSub Topic
@@ -533,8 +533,8 @@ jobs:
         if: always()
         run: |
           echo "Deleting objects in GCS Bucket: ${{ env.GCS_BUCKET }}/temp/"
-          gsutil -m rm -r "${{ env.GCS_BUCKET }}/temp/leaderboard/**" || echo "Failed to delete objects in GCS leaderboard temp folder."
-          gsutil -m rm -r "${{ env.GCS_BUCKET }}/temp/gamestats/**" || echo "Failed to delete objects in GCS gamestats temp folder."
+          gcloud storage rm -r "${{ env.GCS_BUCKET }}/temp/leaderboard/**" || echo "Failed to delete objects in GCS leaderboard temp folder."
+          gcloud storage rm -r "${{ env.GCS_BUCKET }}/temp/gamestats/**" || echo "Failed to delete objects in GCS gamestats temp folder."
           echo "Removing local log and jobid files..."
           rm -f leaderboard_dataflow_submit.log gamestats_dataflow_submit.log injector_run.log
           rm -f leaderboard_dataflow_jobid.txt # Remove Leaderboard jobid file here

.github/workflows/run_rc_validation_python_yaml.yml

@@ -265,9 +265,9 @@ jobs:
           sleep 60
           # Check if any files matching the pattern exist within the unique output folder.
           echo "Checking for files matching pattern: ${OUTPUT_PATTERN}"
-          if gsutil ls "${OUTPUT_PATTERN}" > /dev/null 2>&1; then
+          if gcloud storage ls "${OUTPUT_PATTERN}" > /dev/null 2>&1; then
             echo "SUCCESS: Found output files matching pattern in GCS."
-            gsutil ls "${OUTPUT_PATTERN}" # List found files
+            gcloud storage ls "${OUTPUT_PATTERN}" # List found files
           else
             echo "ERROR: No output files found matching pattern '${OUTPUT_PATTERN}' in GCS bucket."
             exit 1
@@ -280,7 +280,7 @@ jobs:
         run: |
           echo "Deleting unique run folder in GCS: ${GCS_UNIQUE_FOLDER_PREFIX}"
           # Delete the entire unique folder for this run, including temp, staging, and output
-          gsutil -m rm -r "${GCS_UNIQUE_FOLDER_PREFIX}" || echo "Failed to delete unique run folder ${GCS_UNIQUE_FOLDER_PREFIX} in GCS. Manual cleanup might be required."
+          gcloud storage rm -r "${GCS_UNIQUE_FOLDER_PREFIX}" || echo "Failed to delete unique run folder ${GCS_UNIQUE_FOLDER_PREFIX} in GCS. Manual cleanup might be required."
 
           echo "Removing local log, yaml, and jobid files..."
           rm -f yaml_dataflow_submit.log ${{ env.YAML_PIPELINE_FILE }} yaml_dataflow_jobid.txt

.test-infra/dataproc/flink_cluster.sh

@@ -91,7 +91,7 @@ YARN_APPLICATION_MASTER=""
 
 function upload_init_actions() {
   echo "Uploading initialization actions to GCS bucket: $GCS_BUCKET"
-  gsutil cp -r $INIT_ACTIONS_FOLDER_NAME/* $GCS_BUCKET/$INIT_ACTIONS_FOLDER_NAME
+  gcloud storage cp -r $INIT_ACTIONS_FOLDER_NAME/* $GCS_BUCKET/$INIT_ACTIONS_FOLDER_NAME
 }
 
 function get_leader() {

.test-infra/metrics/build.gradle

@@ -54,6 +54,7 @@ composeUp {
   dependsOn "createEmptyConfig"
 }
 dockerCompose {
+  projectName = 'beammetrics'
   environment.put 'DOCKER_CONFIG', project.rootProject.buildDir
 }
 

.test-infra/metrics/influxdb/Dockerfile

@@ -16,14 +16,13 @@
 # limitations under the License.
 ################################################################################
 
-FROM python:3.10-slim
-
-RUN pip install --no-cache-dir gsutil
+FROM alpine:latest
 
 WORKDIR /
 
-RUN gsutil cp gs://apache-beam-testing-metrics/influxdb-backup.tar.gz . && \
-tar xzf influxdb-backup.tar.gz
+RUN apk add --no-cache curl tar && \
+    curl -sS https://storage.googleapis.com/apache-beam-testing-metrics/influxdb-backup.tar.gz -o influxdb-backup.tar.gz && \
+    tar xzf influxdb-backup.tar.gz
 
 
 FROM influxdb:1.8.0

.test-infra/metrics/kubernetes/beam-influxdb-autobackup.yaml

@@ -47,9 +47,12 @@ spec:
                 - mountPath: /backup
                   name: shared-data
             - name: copy-to-gsc-bucket
-              image: gcr.io/apache-beam-testing/gsutil
+              image: gcr.io/google.com/cloudsdktool/google-cloud-cli:stable-slim
+              env:
+                - name: CLOUDSDK_CORE_PROJECT
+                  value: apache-beam-testing
               command: ['sh', '-c', 'tar czf influxdb-backup.tar.gz /backup
-                        && gsutil cp influxdb-backup.tar.gz
+                        && gcloud storage cp influxdb-backup.tar.gz
                         gs://apache-beam-testing-metrics/']
               volumeMounts:
                 - mountPath: /backup

examples/java/src/main/java/org/apache/beam/examples/complete/game/UserScore.java

@@ -214,7 +214,7 @@ public interface Options extends PipelineOptions {
     day's worth (roughly) of data.
 
     Note: You may want to use a small sample dataset to test it locally/quickly : gs://apache-beam-samples/game/small/gaming_data.csv
-    You can also download it via the command line gsutil cp gs://apache-beam-samples/game/small/gaming_data.csv ./destination_folder/gaming_data.csv */
+    You can also download it via the command line gcloud storage cp gs://apache-beam-samples/game/small/gaming_data.csv ./destination_folder/gaming_data.csv */
     @Default.String("gs://apache-beam-samples/game/gaming_data*.csv")
     String getInput();
 

examples/multi-language/README.md

@@ -124,7 +124,7 @@ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.multilanguage.Sk
 the digit. The second item is the predicted label of the digit.
 
 ```
-gsutil cat gs://$GCP_BUCKET/multi-language-beam/output*
+gcloud storage cat gs://$GCP_BUCKET/multi-language-beam/output*
 ```
 
 #### Instructions for running the Java pipeline at HEAD (Beam 2.41.0 and 2.42.0).
@@ -171,7 +171,7 @@ export GCP_REGION=<GCP region>
 export EXPANSION_SERVICE_PORT=<PORT>
 
 # This removes any existing output.
-gsutil rm gs://$GCP_BUCKET/multi-language-beam/output*
+gcloud storage rm gs://$GCP_BUCKET/multi-language-beam/output*
 
 ./gradlew :examples:multi-language:sklearnMinstClassification --args=" \
 --runner=DataflowRunner \
@@ -188,7 +188,7 @@ gsutil rm gs://$GCP_BUCKET/multi-language-beam/output*
 of the digit. The second item is the predicted label of the digit.
 
 ```
-gsutil cat gs://$GCP_BUCKET/multi-language-beam/output*
+gcloud storage cat gs://$GCP_BUCKET/multi-language-beam/output*
 ```
 
 ### Python Dataframe Wordcount