[Bug] protobuf dependency too lenient

[vglocus]

Is this a new bug in dbt-common?

• I believe this is a new bug in dbt-common
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

As of #219 the new protobuf package dependency is set to ">=5.0,<=6.0".
However some files (e.g. dbt_common/events/types_pb2.py) are generated using a later version than 5.0.

With protobuf you should not run code generated with a later version on an earlier runtime. See cross-version-runtime-guarantee.

For example my environment has a 5.28.1 runtime (which satisfies ">=5.0,<=6.0") but is earlier than 5.28.3 which is what dbt_common/events/types_pb2.py is generated with.

The result is the following error when trying to run dbt_common/events/types_pb2.py

google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading types.proto: gencode 5.28.3 runtime 5.28.1. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

Suggested solution:
Set protobuf package dependency to ">=5.28.3,<=6.0" to allow dependency management of the users to solve this.

Expected Behavior

For dependency management (pip install) to not accept protobuf versions that will not work with dbt_common.

(In this case dbt_common says anything over 5.0 is fine, while really it must be 5.28.3 or later)

Steps To Reproduce

1. Install dbt-common==1.13.0 together with protobuf==5.28.1
2. Run dbt deps

Relevant log output

09:19:37.808  Traceback (most recent call last):
09:19:37.808    File "/home/workbench/.pyenv/versions/sys/bin/dbt", line 5, in <module>
09:19:37.808      from dbt.cli.main import cli
09:19:37.808    File "/home/workbench/.pyenv/versions/sys/lib/python3.10/site-packages/dbt/cli/__init__.py", line 1, in <module>
09:19:37.808      from .main import cli as dbt_cli  # noqa
09:19:37.808    File "/home/workbench/.pyenv/versions/sys/lib/python3.10/site-packages/dbt/cli/main.py", line 14, in <module>
09:19:37.808      from dbt.cli import requires, params as p
09:19:37.808    File "/home/workbench/.pyenv/versions/sys/lib/python3.10/site-packages/dbt/cli/requires.py", line 5, in <module>
09:19:37.808      from dbt_common.clients.system import get_env
09:19:37.808    File "/home/workbench/.pyenv/versions/sys/lib/python3.10/site-packages/dbt_common/clients/system.py", line 20, in <module>
09:19:37.808      from dbt_common.events.functions import fire_event
09:19:37.808    File "/home/workbench/.pyenv/versions/sys/lib/python3.10/site-packages/dbt_common/events/__init__.py", line 1, in <module>
09:19:37.808      from dbt_common.events.base_types import EventLevel
09:19:37.808    File "/home/workbench/.pyenv/versions/sys/lib/python3.10/site-packages/dbt_common/events/base_types.py", line 4, in <module>
09:19:37.808      from dbt_common.events import types_pb2
09:19:37.808    File "/home/workbench/.pyenv/versions/sys/lib/python3.10/site-packages/dbt_common/events/types_pb2.py", line 12, in <module>
09:19:37.808      _runtime_version.ValidateProtobufRuntimeVersion(
09:19:37.808    File "/home/workbench/.pyenv/versions/sys/lib/python3.10/site-packages/google/protobuf/runtime_version.py", line 106, in ValidateProtobufRuntimeVersion
09:19:37.808      _ReportVersionError(
09:19:37.808    File "/home/workbench/.pyenv/versions/sys/lib/python3.10/site-packages/google/protobuf/runtime_version.py", line 47, in _ReportVersionError
09:19:37.808      raise VersionError(msg)
09:19:37.808  google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading types.proto: gencode 5.28.3 runtime 5.28.1. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee

Additional Context

No response

[dbeatty10]

Thanks for raising this @vglocus !

After internal discussion with our engineering team, we've decided to pin the lower bound and add some tests.

[ChenyuLInx]

We need to bump the lower bound of protobuf to the latest version that we use to generate the proto definitions in

• dbt-common
• dbt-adapters
• dbt-core

It would be great to generate the lowest version of the prototype possible in the spec.

[MichelleArk]

We can also add tests that could either be pre-commit hooks or unit tests that ensure the generated proto files use the lowest supported version. This would ensure we don't introduce protos that are out of the supported range of our dependencies. Not a blocker for this work but would be neat to do at the same time.

Example: https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/events/core_types_pb2.py#L12-L20

[Joshua-Powell-Lambda]

I ran into this bug today on dbt-core==1.9.0. I was able to workaround by rolling back dbt-core version to 1.8.8

[userqjin]

I am seeing runtime_version import issue.

dbt-adapters==1.14.4
dbt-common==1.17.0
dbt-core==1.9.0
protobuf==5.26.1

Which version should be compatible with protobuf >5 ?

/opt/homebrew/Cellar/python@3.11/3.11.11/Frameworks/Python.framework/Versions/3.11/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
abnormal/airflow/dags/dbt_dag_utils_test.py:13: in <module>
    from abnormal.airflow.dags.dbt_dag_utils import (
../py/abnormal/airflow/dags/dbt_dag_utils.py:21: in <module>
    from dbt.cli.main import dbtRunner, dbtRunnerResult
../../.venv/lib/python3.11/site-packages/dbt/cli/__init__.py:1: in <module>
    from .main import cli as dbt_cli  # noqa
../../.venv/lib/python3.11/site-packages/dbt/cli/main.py:11: in <module>
    from dbt.adapters.factory import register_adapter
../../.venv/lib/python3.11/site-packages/dbt/adapters/factory.py:9: in <module>
    from dbt_common.events.functions import fire_event
../../.venv/lib/python3.11/site-packages/dbt_common/events/__init__.py:1: in <module>
    from dbt_common.events.base_types import EventLevel
../../.venv/lib/python3.11/site-packages/dbt_common/events/base_types.py:4: in <module>
    from dbt_common.events import types_pb2
../../.venv/lib/python3.11/site-packages/dbt_common/events/types_pb2.py:9: in <module>
    from google.protobuf import runtime_version as _runtime_version
E   ImportError: cannot import name 'runtime_version' from 'google.protobuf'

[userqjin]

Nevermind, looks need to upgrade to at least protobuf==5.28.3

[moseleyi]

I've been getting the same error:

google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading adapter_types.proto: gencode 5.29.1 runtime 5.29.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

all of the sudden after upgrading dbt-adapters