New warnings come from 3 places
1. Warning C5287: Warning comes from json_lib.c from code like
compile_time_assert((int) JSON_VALUE_NULL == (int) JSV_NULL);
2. Warning C5287: Similar warning come from wc_static_assert() from code
in wolfSSL's header file
3. Warning C5286 in WolfSSL code, -enum_value
(i.e multiplying enum with -1)is used
To fix:
- Disable warnings in WolfSSL code, using /wd<num> flag.
- workaround warning for users of WolfSSL, disable
wc_static_assert() with -DWC_NO_STATIC_ASSERT compile flag
- Rewrite some compile_time_assert in json_lib.c to avoid warning.
- add target_link_libraries(vio ${SSL_LIBRARIES}) so that
vio picks up -DWC_NO_STATIC_ASSERT
Fixing the code adding MySQL _0900_ collations as _uca1400_ aliases
not to perform deep initialization of the corresponding _uca1400_
collations.
Only basic initialization is now performed which allows to watch
these collations (both _0900_ and _uca1400_) in queries to
INFORMATION_SCHEMA tables COLLATIONS and
COLLATION_CHARACTER_SET_APPLICABILITY,
as well as in SHOW COLLATION statements.
Deep initialization is now performed only when a collation
(either the _0900_ alias or the corresponding _uca1400_ collation)
is used for the very first time after the server startup.
Refactoring was done to maintain the code easier:
- most of the _uca1400_ code was moved from ctype-uca.c
to a new file ctype-uca1400.c
- most of the _0900_ code was moved from type-uca.c
to a new file ctype-uca0900.c
Change details:
- The original function add_alias_for_collation() added by the patch for
"MDEV-20912 Add support for utf8mb4_0900_* collations in MariaDB Server"
was removed from mysys/charset.c, as it had two two problems:
a. it forced deep initialization of the _uca1400_ collations
when adding _0900_ aliases for them at the server startup
(the main reported problem)
b. the collation initialization code in add_alias_for_collation()
was related more to collations rather than to memory management,
so /strings should be a better place for it than /mysys.
The code from add_alias_for_collation() was split into separate functions.
Cyclic dependency was removed. `#include <my_sys.h>` was removed
from /strings/ctype-uca.c. Collations are now added using a callback
function MY_CHARSET_LOADED::add_collation, like it is done for
user collations defined in Index.xml. The code in /mysys sets
MY_CHARSET_LOADED::add_collation to add_compiled_collation().
- The function compare_collations() was removed.
A new virtual function was added into my_collation_handler_st instead:
my_bool (*eq_collation)(CHARSET_INFO *self, CHARSET_INFO *other);
because it is the collation handler who knows how to detect equal
collations by comparing only some of CHARSET_INFO members without
their deep initialization.
Three implementations were added:
- my_ci_eq_collation_uca() for UCA collations, it compares
_0900_ collations as equal to their corresponding _uca1400_ collations.
- my_ci_eq_collation_utf8mb4_bin(), it compares
utf8mb4_nopad_bin and utf8mb4_0900_bin as equal.
- my_ci_eq_collation_generic() - the default implementation,
which compares all collations as not equal.
A C++ wrapper CHARSET_INFO::eq_collations() was added.
The code in /sql was changes to use the wrapper instead of
the former calls for the removed function compare_collations().
- A part of add_alias_for_collation() was moved into a new function
my_ci_alloc(). It allocates a memory for a new charset_info_st
instance together with the collation name and the comment using a single
MY_CHARSET_LOADER::once_alloc call, which points to my_once_alloc()
in the server.
- A part of add_alias_for_collation() was moved into a new function
my_ci_make_comment_for_alias(). It makes an "Alias for xxx" string,
e.g. "Alias for utf8mb4_uca1400_swedish_ai_ci" in case of
utf8mb4_sv_0900_ai_ci.
- A part of the code in create_tailoring() was moved to
a new function my_uca1400_collation_get_initialized_shared_uca(),
to reuse the code between _uca1400_ and _0900_ collations.
- A new function my_collation_id_is_mysql_uca0900() was added
in addition to my_collation_id_is_mysql_uca1400().
- Functions to build collation names were added:
my_uca0900_collation_build_name()
my_uca1400_collation_build_name()
- A shared function function was added:
my_bool
my_uca1400_collation_alloc_and_init(MY_CHARSET_LOADER *loader,
LEX_CSTRING name,
LEX_CSTRING comment,
const uca_collation_def_param_t *param,
uint id)
It's reused to add _uca1400_ and _0900_ collations, with basic
initialization (without deep initialization).
- The function add_compiled_collation() changed its return type from
void to int, to make it compatible with MY_CHARSET_LOADER::add_collation.
- Functions mysql_uca0900_collation_definition_add(),
mysql_uca0900_utf8mb4_collation_definitions_add(),
mysql_utf8mb4_0900_bin_add() were added into ctype-uca0900.c.
They get MY_CHARSET_LOADER as a parameter.
- Functions my_uca1400_collation_definition_add(),
my_uca1400_collation_definitions_add() were moved from
charset-def.c to strings/ctype-uca1400.c.
The latter now accepts MY_CHARSET_LOADER as the first parameter
instead of initializing a MY_CHARSET_LOADER inside.
- init_compiled_charsets() now initializes a MY_CHARSET_LOADER
variable and passes it to all functions adding collations:
- mysql_utf8mb4_0900_collation_definitions_add()
- mysql_uca0900_utf8mb4_collation_definitions_add()
- mysql_utf8mb4_0900_bin_add()
- A new structure was added into ctype-uca.h:
typedef struct uca_collation_def_param
{
my_cs_encoding_t cs_id;
uint tailoring_id;
uint nopad_flags;
uint level_flags;
} uca_collation_def_param_t;
It simplifies reusing the code for _uca1400_ and _0900_ collations.
- The definition of MY_UCA1400_COLLATION_DEFINITION was
moved from ctype-uca.c to ctype-uca1400.h, to reuse
the code for _uca1400_ and _0900_ collations.
- The definitions of "MY_UCA_INFO my_uca_v1400" and
"MY_UCA_INFO my_uca1400_info_tailored[][]" were moved from
ctype-uca.c to ctype-uca1400.c.
- The definitions/declarations of:
- mysql_0900_collation_start,
- struct mysql_0900_to_mariadb_1400_mapping
- mysql_0900_to_mariadb_1400_mapping
- mysql_utf8mb4_0900_collation_definitions_add()
were moved from ctype-uca.c to ctype-uca0900.c
- Functions
my_uca1400_make_builtin_collation_id()
my_uca1400_collation_definition_init()
my_uca1400_collation_id_uca400_compat()
my_ci_get_collation_name_uca1400_context()
were moved from ctype-uca.c to ctype-uca1400.c and ctype-uca1400.h
- A part of my_uca1400_collation_definition_init()
was moved into my_uca0520_builtin_collation_by_id(),
to make functions smaller.
MDEV-21978 introduces more preferrable alternatives to those extensions.
This commit removes the above old and deprecated syntax.
With the code for the old extension formats gone,
this commit also improves the code around the new extension suffixes:
* Remove code for formatting ``%`T``
* Those code are now dead,
for the new suffix-based syntax does not recognize `%sQT`/`%sTQ`.
* `suffix_q= TRUE` now additionally replaces `…|= ESCAPED_ARG`.
* Flatten `flag= true if /%iE/` and `do_iE if flag` code together
[Breaking] This commit removes obsolete features. Although my earlier
work (should have) migrated every usages direct or indirect in the
entire MariaDB/server, other codebases might still be using them. This
final change will break *everything* in those outdated foreign lands.
… and delete `%uU` (just use `%d` for that)
The follow-up #3360 discovered `%M` usages that suggest that it was
designed for `errno` and similar **signed** `int` (“errno_t”) variables.
Besides convenience, if the old `%M` read a `signed int`,
so should the new version to maintain compatibility.
I only added `%iE` (no `%dE`) to keep the new suffix mechanics
away from the popular `%d`. Beïng synonyms (originally),
this decision on preserving `%d` also saves the need for `%iI`/`%dD`.
This is the first part of
MDEV-21978 make my_vsnprintf to use gcc-compatible format extensions,
which adds these alternatives to the MySQL extensions.
There’s also the escapes `%sS` & `%uU` for any hippies needing them.
These suffixes are compatible with the C standard and
therefore as well as `printf` tools such as GCC checks.
The old extension formats (e.g., `%M`) are now effectively deprecated,
although they’re left intact for now. For a more sequential
MDEV-21978 process, a separate commit will delete them after we migrate
all `my_vsnprintf` usages to the new preferred syntax. The service’s
major version bumped nonetheless for the new significance of suffixes.
[Breaking] This commit may fail
* on places needing the aforementioned escapes
* because of the major version bump
Reviewed-by: Andrew Hutchings <andrew@mariadb.org>
Reviewed-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
This commit prepares for MDEV-21978 which wants to migrate
extensions from *specifiers* to standard-compatible *suffixes*,
complete with boolean local vars. In this new format,
branches will land on `%s`/`%u` before processing the suffix.
This commit contains non-breaking portions of MDEV-21978 work.
This commit also changes an `if`-`else` chain to
a `switch` block simply because it’s an eyesore.
* rpl.rpl_system_versioning_partitions updated for MDEV-32188
* innodb.row_size_error_log_warnings_3 changed error for MDEV-33658
(checks are done in a different order)
in _ma_unique_hash, skip_trailing_space, my_hash_sort_mb_nopad_bin and my_strnncollsp_utf8mb4_bin
UBSAN detected the nullptr-with-offset in a few places
when handling empty blobs.
Fix:
- Adding DBUG_ASSERT(source_string) into all hash_sort() implementations
to catch this problem in non-UBSAN debug builds.
- Fixing mi_unique_hash(), mi_unique_comp(),
_ma_unique_hash(), _ma_unique_comp() to replace NULL pointer to
an empty string ponter..
Note, we should also add DBUG_ASSERT(source_string != NULL) into
all implementations of strnncoll*(). But I'm afraid the patch
is going to be too long and too dangerous for 10.5.
Also fixes:
MDEV-32190 Index corruption with unique key and nopad collation (without DESC or HASH keys)
MDEV-28328 Assertion failures in btr0cur.cc upon INSERT or in row0sel.cc afterwards
The code in strings/strcoll.inl when comparing an empty string
to a string like 0x0001050001 did not take into account
that the leftmost weight in the latter can be zero, while
there are some more weights can follow the zero weight.
Rewriting the code to treat the shorter string as smaller than
a longer string.
Rewriting loops in check_rules() and init_weight_level()
in the way to avoid UBSAN nullptr-with-offset error.
No MTR test are needed - the reported failures disappeared
from "mtr" output when running an UBSAN compiled build.
With UBSAN builds the function my_string_repertoire_8bit() failed on
"runtime error: applying zero offset to null pointer" when
NULL wad passed as the str parameter.
Fix:
test str for NULL, and return MY_REPERTOIRE_ASCII if str is NULL.
MTR:
This problem made MTR tests
- main.xa_sync
- innodb.xa_debug
- main.xa
fail with the nullptr-with-offset UNSAN error.
After this commit these tests do not fail anymore.
This commit does not need any new MTR tests.
The functions MY_CHARSET_HANDLER::caseup() and MY_CHARSET_HANDLER::casedn()
in their virtual imlementations do "const char *end= src + srclen"
in the very beginning. Therefore src cannot be NULL to avoid
"UBSAN: SUMMARY: UndefinedBehaviorSanitizer: nullptr-with-offset".
Adding DBUG_ASSERT(src != NULL) into all virtual implementations,
to catch this problem in regular Debug builds (without UBSAN).
Fixing Master_info_index::get_master_info() to check connection_name->str.
If it is NULL then passing empty_clex_str into IdentBufferCasedn
instead of *connection_name.
The problem was that get_collation_number_internal() loops over all
collations for finding a collation based on name. For looking up
utf8mb4_0900_ aliases it used 22633 character strings comparisons at
startup.
Fixed by adding the MariaDB internal collation number in the "0900" alias
lookup array. This is fine as collation numbers never changes.
Discussed-with: serg@mariadb.com
This is done by mapping most of the existing MySQL unicode 0900 collations
to MariadB 1400 unicode collations. The assumption is that 1400 is a super
set of 0900 for all practical purposes.
I also added a new function 'compare_collations()' and changed most code
to use this instead of comparing character sets directly.
This enables one to seamlessly mix-and-match the corresponding 0900 and
1400 sets. Field comparision and alter table treats the character sets
as identical.
All MySQL 8.0 0900 collations are supported except:
- utf8mb4_ja_0900_as_cs
- utf8mb4_ja_0900_as_cs_ks
- utf8mb4_ru_0900_as_cs
- utf8mb4_zh_0900_as_cs
These do not have corresponding entries in the MariadB 01400 collations.
Other things:
- Added COMMENT colum to information_schema.collations. For utf8mb4_0900
colletions it contains the corresponding alias collation.
strerror_s on Linux will, for unknown error codes, display
'Unknown error <codenum>' and our tests are written with this assumption.
However, on macOS, sterror_s returns 'Unknown error: <codenum>' in the
same case, which breaks tests. Make my_strerror consistent across the
platforms by removing the ':' when present.
During a query execution some sorting and grouping operations
on strings may be involved. System variable max_sort_length defines
the maximum number of bytes to use when comparing strings during
sorting/grouping. Thus, the comparable parts of strings may be less
than their actual size, so the results of the query may be not
sorted/grouped properly.
To indicate that some comparisons were done on a truncated lengths,
a new warning has been introduced with this commit.
Step#1: fixing the return type of strnxfrm() from size_t to this structure:
typedef struct
{
size_t m_output_length;
size_t m_source_length_used;
uint m_warnings;
} my_strnxfrm_ret_t;
