Most BCP improvements from NCBI, as of May 2024

[ucko]

This PR roughly corresponds to the primarily BCP-related commits from #555, adding 02aa63c per dependency considerations and omitting a couple of commits that can receive individual PRs, which I'll open shortly.

[ucko]

I've force-pushed a corrected version superseding #570.

[ucko]

Meanwhile, I'm looking to extend this series with BCP-specific tuneups superseding #573. To that end, I wanted to compare behavior for three cases: normal BCP in, BCP into an alternate version of the table where column types are all VARCHAR(64) (to confirm that conversion to VARCHAR(n) is OK when truncation definitely won't be needed), and BCP into an alternate version where they're all VARCHAR(1) (to see what happens when truncation definitely will be needed), using ctlib/unittests/blk_in.c and dblib/unittests/bcp.c as starting points.

On the DBLIB front, I found that all three cases fully succeeded when built against this branch, but VARCHAR(1) failed for at least one column (details TBD) when using proprietary libraries, so more strictness may actually be in order. On the CTLIB front, something appears to be off altogether, as even the initial run with the usual column types fails with

Open Client Message: 0x608010 0x6259b0
number 0x1010104 layer 1 origin 1 severity 1 number 4
msgstring: blk_describe(): blk layer: user error: Parameter colnum has an illegal value of 1
osstring: (null)
blk_describe(1) failed
Open Client Message: 0x608010 0x6259b0
number 0x1010104 layer 1 origin 1 severity 1 number 4
msgstring: blk_describe(): blk layer: user error: Parameter colnum has an illegal value of 2
osstring: (null)
blk_describe(2) failed
[...]

Any idea what I might be missing? I took care to predefine SYB_LP64 and verified that only FreeTDS-specific headers come from FreeTDS.

UPDATE: Those errors turned out to stem from the use of CS_VERSION_100 and BLK_VERSION_100; advancing the requested version helped.

[ucko]

Here are my summarized comparison results (comparing against version 16.0 of Sybase's client libraries), where *B is (VAR)BINARY; *C is (VAR)CHAR; B, C, VB, and VC are respectively BINARY, CHAR, VARBINARY, and VARCHAR; *DT is (SMALL)DATETIME; *$ is (SMALL)MONEY; and # is any numerical type -- BIT, any variant of INT, REAL, FLOAT, DECIMAL, or NUMERIC:

CTLIB BCP truncation differences w/CS layer messages ignored (see below for corrected version):

FROM\TO	*B(1)	*B(64)	*C(1)	*C(64)
C(10)	a	a	b	OK
VC(10)	a	a	c	OK
B(10)	b	OK	c/d	OK
VB(10)	e	OK	e	OK
*DT	f/g	g	c	OK
*$/#	c/h	OK	e/h	OK

• a)
  • Sybase: clearly expects hex digits; behavior depends on what it sees first:
    • non-hex char among first 2n: rowxfer fails w/syntax error
    • excess char when length is implicit [C(n)]: xfer noisily truncates but OK(!)
    • excess char when length is given [VC(n)]: rowxfer fails w/overflow error
  • FreeTDS: rowxfer silently fails in all those cases
• b) silently truncates under Sybase (but OK), fails under FreeTDS
• c) noisily truncates in rowxfer under Sybase (but OK), fails under FreeTDS
• d) B(n) -> *C(m): silently truncates under Sybase when 4 <= m < 2n; the cutoff of 4 suggests a leading 0x, but VB(n) -> *C(2n) is OK
• e) rowxfer fails under both, (overflow) error only under Sybase
• f) S: bind fails (type mismatch), xfer OK (!); F: xfer fails; (g) if NULL
• g) rowxfer always OK, but Sybase bind fails (type mismatch)
• h) BIT is always OK, as is TINYINT -> *BINARY(1)

DBLIB BCP truncation differences (also corrected below):

FROM\TO	*B(1)	*B(64)	*C(1)	*C(64)
*C(10)	a	a	b	OK
B(10)	c	OK	b/d	OK
VB(10)	b	OK	b/d	OK
*DT/*$	b	OK	b	OK
#	b/e	OK/f	b/e	OK/f

• a)
  • Sybase: syntax error if non-hex char among first 2n, otherwise overflow error as appropriate
  • FreeTDS: syntax error if non-hex char present at all, otherwise silently truncates
• b) conversion overflow error (20049) from Sybase, silent truncation by FreeTDS
• c) oversized row error (20074) from Sybase, silent truncation by FreeTDS
• d) *B(n) -> *C(m): silently truncates under Sybase when 4 <= m < 2n
• e) BIT is always OK, as is TINYINT -> *BINARY(1)
• f) Sybase crashes with heap corruption for DECIMAL and NUMERIC

[ucko]

I'll turn back to other PRs for now, but do still plan to add a proper replacement for #573 at some point.

[ucko]

Ah, blk_in (like most ctlib unit tests) doesn't register a handler for messages from the CS layer, just from the CT layer and the server; it looks like resurveying with one in place will make a difference, at least when running against FreeTDS.

[ucko]

CTLIB BCP truncation differences w/CS layer messages printed, and more corner cases examined

General notes

• This behavior is consistent between 15.7 and 16.0 libraries when using {CS,BLK}_VERSION_125 and up.
  • blk_describe fails altogether under {CS,BLK}_VERSION_120 and below.
  • blk_rowxfer started allowing noisy truncation [(1, 2, 1, 42)] sometime between 12.5 (which issued that same error but then failed) and 15.7; I don't have any installations with intermediate versions, but even 16.0 is a decade old. (At any rate, all other cases, including those that encounter silent truncation, behave the same across versions, modulo 12.5's more limited data type support.)
• LC and LB are LONGCHAR and LONGBINARY, which act like base versions; L?C and L?B are short for C/LC and B/LB respectively; *DT now additionally covers DATE, TIME, and BIG(DATE)TIME; L and USH are LONG and USHORT, which do not act like *INT.
• UNICHAR is out of scope per broader differences -- Sybase expects even-length input, which it inserts into binary columns as is modulo possible truncation (but does attempt to convert when inserting into string columns), whereas FreeTDS treats it like (LONG)CHAR.
• DECIMAL and NUMERIC also work differently, but only in the context of binary columns -- Sybase stores just the main body (and drops trailing NUL bytes as usual), whereas FreeTDS stores the whole thing (space permitting), complete with trailing NUL bytes per its usual practice.

FROM\TO	*B(1)	*B(64)	*C(1)	*C(64)
L?C(10)	a	a	b	OK
VC(10)	a	a	c	OK
L?B(10)	b	OK	c	OK
VB(10)	d	OK	d	OK
*DT	e/f	f	c	g
*$/#	c/h	OK	d/h	i
L/USH	j	j	j	j

• a) Both expect hex digits with optional leading 0x (and effectively prepend a 0 to the body if the length is odd), but their behavior when something is off otherwise varies.
  • Sybase: clearly expects hex digits; behavior depends on what it sees first:
    • non-hex char among first 2n(+2): rowxfer fails w/syntax error
      (1, 2, 1, 26) - blk_rowxfer(): blk layer: internal BLK-Library error: Failed in conversion routine - syntax error. col = # row = #.
    • excess char w/CS_NULLTERM [C(n)/LC(n)]: xfer noisily truncates but OK(!); if the accompanying full length is odd, effectively prepends a zero and discards the last digit even if the initial run of hex digits has even length.
      (1, 2, 1, 42) - blk_rowxfer(): blk layer: internal BLK-Library error: Data truncated while doing local character set conversion. col = # row = #.
    • excess char when length is restated [VC(n)]: rowxfer fails w/overflow error
      (1, 2, 1, 25) - blk_rowxfer(): blk layer: internal BLK-Library error: Failed in conversion routine - condition overflow. col = # row = #.
  • FreeTDS:
    • any other syntax anywhere: rowxfer fails w/syntax error
      (2, 4, 1, 24) - cs_convert: cslib user api layer: common library error: The conversion/operation was stopped due to a syntax error in the source field.
    • excess char otherwise: rowxfer fails w/overflow error
      (2, 4, 1, 36) - cs_convert: cslib user api layer: common library error: The result is truncated because the conversion/operation resulted in overflow.
• b) silently truncates under Sybase (but OK), fails under FreeTDS -- (2, 4, 1, 36) as above
• c) noisily truncates in rowxfer under Sybase (but OK): (1, 2, 1, 42) as above; NB: truncates to even length when starting with binary
  fails under FreeTDS: (2, 4, 1, 36) as above
• d) rowxfer overflow error - (1, 2, 1, 25) under Sybase, (2, 4, 1, 36) under FTDS
• e)
  • Sybase: bind fails (type mismatch), blk_rowxfer OK (!), storing raw content (quite possibly truncated)
    (1, 1, 1, 10) - blk_bind(): blk layer: user error: a table column of type # cannot be bound to a program variable of type #.
  • FreeTDS: blk_rowxfer fails [(2, 4, 1, 36) as above]; (f) if NULL
• f) rowxfer always OK, but Sybase bind fails (type mismatch as above)
• g) OK, but hour padding differs (space with Sybase, 0 with FreeTDS) and FreeTDS prepends Jan 01 1900 for pure (BIG)TIME values and appends 12:00AM for pure DATE values
• h) BIT is always OK, as is (U)TINYINT -> *BINARY(1)
• i) OK, but only Sybase trims trailing zeroes from fractional MONEY(4) values; OTOH, only Sybase formats CS_REAL (float) values with excess precision.
• j) Same as (e) [length 1, not NULL]/(f) for Sybase; under FreeTDS, rowxfer fails with a type mismatch (unless NULL)
  (2, 1, 1, 16) - cs_convert: cslib user api layer: external error: Conversion between # and # datatypes is not supported.

Notably, all errors under Sybase officially come from the CT/BLK layer, whereas FreeTDS only ever issues CS-layer errors that offer less context and may be likelier to slip through the cracks. That should be possible to remedy by letting _cs_convert take an optional pointer to a BCP-error variable and populate it in lieu of reporting an error directly; this pointer's presence would also result in behavior tweaks.

[ucko]

Corrected DBLIB truncation differences (modulo formatting details along the above lines and FreeTDS's support for additional client types):

FROM\TO	*B(1)	*B(64)	*C(1)	*C(64)
*C(10)	a	a	b	OK
*B(10)	b	OK	c	OK
*DT/*$	c	OK	c	OK
#	c/d	OK/e	c/d	OK/e

• a) Both expect hex digits with optional leading 0x (and effectively prepend a 0 to the body if the length is odd), but their behavior when something is off otherwise varies.
  • Sybase: syntax error (20050) if non-hex char among first 2n(+2), otherwise overflow error (20049) as appropriate
  • FreeTDS: syntax error (20050) if non-hex char present at all, otherwise silently truncates
• b) oversized row error (20074) from Sybase, silent truncation by FreeTDS
• c) overflow error (20049) from Sybase, silent truncation by FreeTDS
• d) BIT is always OK, as is TINYINT -> *BINARY(1)
• e) Sybase crashes with heap corruption for DECIMAL and NUMERIC

[ucko]

My latest version of this series adds a commit to address a corner case I recently encountered by (re)allowing BCP into computed columns in the context of a FIRE_TRIGGERS hint, since an INSTEAD OF INSERT trigger may well be in play.