Skip to content

Fix DB2 LUW reflection, binary binds and XML/DECFLOAT results - #205

Open
aminghadersohi wants to merge 1 commit into
ibmdb:masterfrom
aminghadersohi:fix-luw-reflection-and-binds
Open

aminghadersohi wants to merge 1 commit into
ibmdb:masterfrom
aminghadersohi:fix-luw-reflection-and-binds

Conversation

@aminghadersohi

Copy link
Copy Markdown

Problems (found against a local DB2 LUW 11.5.9.0 server)

Reflection (DB2Reflector, LUW only):

  • Foreign keys leak across schemas. get_foreign_keys filters the constrained table by name only, so for A.CHILD it also returns the foreign keys of B.CHILD. Columns of a composite key are not paired by key position.
  • Column names split. get_pk_constraint and get_indexes split SYSCAT.INDEXES.COLNAMES on \w+, so a column "AMT$X" is reflected as two columns amt and x. DESC index columns lose their ordering. The primary key reports its backing index name (SQL2609...) instead of the constraint name.
  • Phantom indexes. DB2's internal XML path index on an XML column is reflected as a user index.
  • Unique constraints are not ordered by key position.
  • Inspector.has_sequence raises TypeError: has_sequence() got an unexpected keyword argument 'info_cache'.
  • get_check_constraints is not implemented.
  • Types. DECFLOAT, BINARY and VARBINARY reflect as NullType with a warning; CHAR/VARCHAR FOR BIT DATA reflect as CHAR/VARCHAR although the DBAPI returns bytes.

ibm_db dialect:

  • Binary executemany corrupts data. SQLAlchemy binds binary values through dbapi.Binary, which ibm_db_dbi implements as memoryview. In executemany, ibm_db rejects a memoryview for BINARY/VARBINARY/FOR BIT DATA (SQL0302N) and stores its repr text (<memory at 0x...>) in BLOB columns. Single-row execute is fine. Reproduced with ibm_db 3.2.3 and 3.3.0. Binding bytes works in both paths.
  • DECFLOAT results are str; they are now Decimal (or float with asdecimal=False), and Decimal binds are sent as exact text.
  • XML results start with a byte order mark and a <?xml version="1.0" encoding="UTF-16" ?> declaration synthesized by the CLI (even for a document stored with a UTF-8 declaration). DB2 stores XML without a declaration, so exactly that prefix is dropped.
  • Disconnects. A connection lost while fetching surfaces as the base ibm_db_dbi.Error, which is_disconnect ignored, so the dead connection went back to the pool. After a server restart, close() on a dead connection raises CLI0106E, and the pool logged an error for every connection it discarded; that one error is now ignored in do_close.

IBM i (AS400Reflector) and z/OS (OS390Reflector) are unchanged. The default-schema issue is handled separately in #203.

Tests

New test/test_luw_reflection.py, same style as the existing tests: connectionless processor/keyword checks and live reflection and round-trip checks (fixtures in two schemas with same-named tables, a composite FK with reversed key order, a "AMT$X" column, a DESC index, an XML column, a check constraint, a sequence, DECFLOAT/BINARY/VARBINARY/FOR BIT DATA/BLOB values via executemany).

Same test file, Python, SQLAlchemy, DBAPI and server; only ibm_db_sa/ differs between upstream master 7f3866e and this patch:

master this patch
test/test_luw_reflection.py 12 FAIL / 1 PASS 13 PASS

(The unique-constraint test passes on master for this fixture; master already joins TABCONST/KEYCOLUSE on schema and table.)

python -m pytest -p sqlalchemy.testing.plugin.pytestplugin --dburi "db2+ibm_db://user:pass@host:port/db" -q test/test_luw_reflection.py

Validation: Python 3.11, SQLAlchemy 2.0.52, ibm-db 3.2.3, pytest 9.1.1, local throwaway DB2 Community Edition 11.5.9.0 container. The full legacy suite was not run.

Reflection on DB2 LUW (DB2Reflector):
- get_foreign_keys filtered the constrained table by name only, so a
  same-named table in another schema contributed its keys, and it did not
  pair columns by key position. Read SYSCAT.REFERENCES with KEYCOLUSE,
  scoped by schema and name, ordered by COLSEQ; report ON DELETE/UPDATE
  rules.
- get_pk_constraint and get_indexes split SYSCAT.INDEXES.COLNAMES on \w+,
  so a column such as "AMT$X" became two columns and DESC was lost; the
  primary key reported its backing index name. Read KEYCOLUSE/INDEXCOLUSE
  instead, report column_sorting, and skip DB2's internal XML region and
  path indexes.
- get_unique_constraints now orders columns by key position.
- get_check_constraints is implemented from SYSCAT.CHECKS.
- has_sequence accepts the keywords SQLAlchemy 2's Inspector passes
  (info_cache), which raised TypeError.
- get_columns reflects DECFLOAT, BINARY and VARBINARY (previously NullType)
  and CHAR/VARCHAR FOR BIT DATA as BINARY/VARBINARY, matching the bytes the
  DBAPI returns.

ibm_db dialect:
- Binary values are bound as bytes. dbapi.Binary is a memoryview, which
  ibm_db's executemany rejects for BINARY/VARBINARY/FOR BIT DATA (SQL0302N)
  and stores as its repr text in BLOB columns.
- DECFLOAT results are returned as Decimal (the DBAPI returns str).
- XML results drop the byte order mark and UTF-16 declaration the CLI adds
  when serializing.
- is_disconnect recognizes a lost connection reported as the base
  ibm_db_dbi.Error (raised while fetching), and closing an already-closed
  connection no longer raises during pool cleanup.

IBM i and z/OS reflectors are unchanged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant