* include/wget/wget.h: Make library symbol export attribute
((__visibility__)) preceed the function prototype to work on Windows
with MSVC
* tests/libtest.h: Same
Reported-by: Gisle Vanem
* configure.ac: Suppress -Wunsuffixed-float-constants for gcc,
add -Wno-disabled-macro-expansion to clang flags
* examples/getstream.c: Removed 'unused' attributes from argc and argv.
* include/wget/wget.h: Fix slot to int for bar functions.
* libwget/bar.c: Fix slot to int for bar functions, cast to int,
change type of bitfield 'redraw' to unsigned.
* libwget/hsts.c (wget_hsts_new): Fix int to bitfield conversion.
* libwget/http.c: Cast printf %p format to (void *).
* src/wget.c: Cast printf %p format to (void *),
remove ACTION_DONE from enum actions.
* src/wget_host.h: Fix double declaration of host_remove_job()
* tests/libtest.c (wget_test_check_filesystem): Fix type of 'rc'.
* tests/test-idn-robots.c: Fix comma to semikolon at end of line.
* tests/test.c (test_strcasecmp_ascii): Fix type of loop variable.
* src/options.c: Add new --header option and a new parse_header()
parsing function for that. Also add self tests for parsing
* src/wget_options.h: Same
* src/wget.c (http_create_request): Add the user added headers to the
HTTP Request
* src/test-wget-1.c: Add a test for the --header option
* .travis.sh: Rewrite for clarity and speed. Testing shows that valgrind
and ASan don't play well together. So split their executions.
* .travis.yml: Enable ccache for builds. Also try to update libc6 if
possible
* .travis.yml: Do not run on OSX with CC=gcc. This is because on OSX,
gcc is only a wrapper around clang.
* .travis_setup.sh: Cosmetic changes only
* Makefile.am (check-coverage): Explcicitly disable valgrind tests when
checking coverage
* tests/libtest.c (wget_test): Use a provided suppressions file for
valgrind. This is required since the valgrind version on Travis seems to
be out of sync with libc. This causes valgrind to report a memory bug
falsely.
* tests/valgrind-suppressions: New file with some valgrind suppressions.
Included suppressions are for a valgrind/libc bug and to suppress
reachable memory messages from within gnutls
* include/wget/wget.h: Declare new function wget_determine_screen_width.
Also change prototype of wget_bar_init to not accept a max_width
parameter
* src/utils.c: Move function determine_screen_width from here ...
* libwget/utils.c: ... to here.
* src/utils.{c,h}: Delete empty file
* src/Makefile.am: Remove source files utils.{c,h}
* src/bar.c: Move detection of screen width from here ...
* libwget/bar.c: ... to here
* tests/test.c: wget_bar_init() no longer takes a max_width parameter
* libwget/iri.c (wget_iri_unescape_inline): Do not touch '+'.
(wget_iri_parse): Replace '+' by ' ' in query part.
* tests/test.c (test_iri_parse): Add more tests
* include/libwget.h.in: Add function wget_list_getnext().
* libwget/list.c: Add function wget_list_getnext().
* libwget/robots.c: Fix memory leak.
* src/host.c (host_remove_job): Cleanup queue after downloading and
scanning robots.txt.
* src/job.h (struct JOB): Add flag 'requested_by_user'.
* src/wget.c (add_url_to_queue): Set 'requested_by_user',
(add_url): Fix checking for disallowed paths.
* tests/Makefile.am: Add test 'test-robots'.
* tests/test-robots.c: New test to prove robots functionality.
Special handling for automatic robots.txt jobs
==============================================
What can happen with --recursive and --span-hosts is that a document from hostA
has links to hostB. All these links might go into the hostB queue before robots.txt
is downloaded and parsed. To avoid downloading of 'disallowed' documents, the queue
for hostB has to be cleaned up right after downloading and parsing robots.txt.
Any links links that have been explicitly requested by the user are still downloaded.
* libwget/cookie.c (_wget_cookie_normalize_cookie): Add checks
for __Secure- and __Host- prefixes.
* tests/test.c (test_cookies): Add test cases for the prefixes.
This change implements proposals from draft-ietf-httpbis-cookie-prefixes-00
as modern browsers do.
* tests/test-wget-1.c: Add new test. If the remote filename
exists locally, then Wget2 tries to conitnue downloading that file
despite being given an explicit output document. This test checks
for such behaviour.
* tests/libtest.c: New function wget_test_check_filesystem(),
(_empty_directory, _remove_directory): Do not print
errors about missing directories.
* tests/libtest.h: Add define WGET_TEST_FS_CASEMATTERS
* tests/test--accept.c: Renamed files to avoid casing problems
* tests/test-wget-1.c: Skip casing tests on case-sensitive filesystem
* src/options.c: Add --content-on-error to options and to help text
* src/options.h: Add content_on_error to struct config
* src/wget.c: Check config.content_on_error
* tests/test-wget-1.c: Add tests for --content-on-error
* tests/Makefile.am: Add $(EXEEXT) to executable names
* tests/libtest.c: Add EXEEXT to wget2 executable,
add support for EMULATOR env variable - e.g. to test with wine
* examples/print_html_urls.c (html_parse_localfile):
Detect BOM (Byte Order Mark), convert UTF-16 to UTF-8 before parsing.
* include/libwget.h.in: Remove wget_charset_transcode(),
add wget_memiconv() and wget_striconv().
* libwget/encoding.c: Implement wget_memiconv() and wget_striconv()
* src/wget.c (html_parse): Add 'html_len' param,
convert UTF-16 to UTF-8 before parsing.
* tests/test.c: New test for wget_memiconv().
* tests/Makefile.am: Add test-metalink
* tests/test-metalink.c: New file
This new tests are
- --no-follow-metalink
- metalink v3, without pieces
- metalink v4, without pieces
- metalink v4, with pieces
* include/libwget.h.in: Add sort_age to struct wget_cookie_st
* libwget/cookie.c: Many code enhancements
* libwget/http.c (wget_http_parse_setcookie): Relax parsing,
(wget_http_parse_setcookie): Allow \n and \r\n as EOL
* libwget/private.h: Add missing #includes
* tests/Makefile.am: Add disabled test-cookies-http_state
* tests/test-cookies-http_state.c: New cookie test
There still 30 out of 220 tests failing, before this commit we had
135 failing. The left over tests are mainly corner cases, but feel free
to work on it. When all tests pass, we enable test-cookies-http_state
in Makefile.am.
* src/Makefile.am: Add wget2_noinstall target
* src/wget.c: On exit, only release memory within test suite
* tests/libtest.c: Call wget2_noinstall for testing
* include/libwget.h.in: Include <stdbool.h>.
Define wget_thread_support().
* libwget/thread.c: Change condition to check
for threading support.
(wget_thread_support): New function.
(wget_thread_start): [!USE_POSIX_THREADS && !USE_PTH_THREADS]
Invoke directly the function.
* src/wget.c (main): If no threads are supported then set the
number of max threads to 1. If the queue is not empty, call again the
downloader function. It will be used only in case no threads are
available.
(downloader_thread): Do not wait for new elements in the queue
when there are no threads.
* tests/libtest.c (wget_test_start_server): Skip the test if
there is no threads support.
* .travis.sh: Add --enable-valgrind-tests to ./configure,
export VALGRIND_TESTS=1
* .travis.yml: Install valgridn on Mac OS X
* tests/libtest.c: Add --max-threads=1 --prefer-family=ipv4 to wget2 command
