Commit Graph

861 Commits

Author SHA1 Message Date
ac83eb4a98 * libwget/hashmap_old.c: Removed 2016-11-03 09:58:02 +01:00
f26f23bcd3 * configure.ac: Understand 'Debian clang' and 'gcc-x.y' for 2016-11-03 09:51:25 +01:00
b049eb6d3a Fix gcc/clang warnings from the new manywarnings feature
* configure.ac: Suppress -Wunsuffixed-float-constants for gcc,
  add -Wno-disabled-macro-expansion to clang flags
* examples/getstream.c: Removed 'unused' attributes from argc and argv.
* include/wget/wget.h: Fix slot to int for bar functions.
* libwget/bar.c: Fix slot to int for bar functions, cast to int,
  change type of bitfield 'redraw' to unsigned.
* libwget/hsts.c (wget_hsts_new): Fix int to bitfield conversion.
* libwget/http.c: Cast printf %p format to (void *).
* src/wget.c: Cast printf %p format to (void *),
  remove ACTION_DONE from enum actions.
* src/wget_host.h: Fix double declaration of host_remove_job()
* tests/libtest.c (wget_test_check_filesystem): Fix type of 'rc'.
* tests/test-idn-robots.c: Fix comma to semikolon at end of line.
* tests/test.c (test_strcasecmp_ascii): Fix type of loop variable.
2016-11-02 22:19:41 +01:00
4c6d6179f2 Replace manywarnings module by our own code
* bootstrap.conf: Remove manywarnings, add warnings
* configure.ac: Replace manywarnings code

The new code works on recent gcc and clang with an overhead of
~100ms instead of ~9s here (without -C).
This is meant for developers only and is activated by either
--enable-gcc-warnings or by touching .manywarnings.
2016-11-02 16:51:49 +01:00
f467b8fd86 Fix parse_header() and add more tests
* src/options.c (parse_header): Fix bug where empty string did not clear
out all the headers
* src/options.c (selftest_options): Fix testcase for parse_header and
add more tests
2016-10-23 15:32:22 +02:00
cb5f9a230f Check for gcrypt.h
* configure.ac: Check for gcrypt.h
* libwget/hashfile.c: Check for HAVE_GCRYPT_H
2016-10-17 21:05:23 +02:00
fd46f870b1 Add gnulib digest hash fallback
* bootstrap.conf: Add md2, md5, sha1, sha256, sha512 modules
* libwget/hashfile.c: Add code branch to use gnulib functions if needed
2016-10-17 21:04:36 +02:00
0bcf75bb9b * tests/test-iri.c: Fix test to work on non-UTF-8 locales 2016-10-17 11:41:13 +02:00
0bcd506146 * tests/test-iri-percent.c: Fix test to work on non-UTF-8 locales 2016-10-17 11:41:10 +02:00
bcf1455834 * tests/test-restrict-ascii.c: Fix test to work on non-UTF-8 locales 2016-10-17 11:41:07 +02:00
b701a06495 * libwget/hashfile.c: Remove unused defines 2016-10-17 11:41:01 +02:00
5225421c31 Make --header replace wget's standard headers
* libwget/http.c (wget_http_create_request): Add 'Host' header
  to header list,
  (wget_http_send_request): Do not add 'Host' hard-coded,
  (wget_http_request_to_buffer): Special treatment of 'Content-Length'
* src/wget.c (http_create_request): Replace wget's HTTP headers by
  user-provided headers, except 'Cookie' headers which will be appended.
2016-10-14 12:51:12 +02:00
502b751e54 Support --header option
* src/options.c: Add new --header option and a new parse_header()
parsing function for that. Also add self tests for parsing
* src/wget_options.h: Same
* src/wget.c (http_create_request): Add the user added headers to the
HTTP Request
* src/test-wget-1.c: Add a test for the --header option
2016-10-13 22:37:28 +02:00
861385a6c8 Separate progress bar logic into library and application parts
* include/wget/wget.h: Remove struct _wget_bar_ctx and wget_bat_ctx_t,
  amend prototypes of wget_bar functions,
  new function wget_bar_write_line()
* libwget/bar.c: Bunch of cleanups and code rearrangements
* src/wget_progress.h: Remove
* src/Makefile.am: Remove src/wget_progress.h
* src/bar.c: Remove mutex usage,
  (_error_write) call wget_bar_write_line
* src/log.c: Remove redundant param checks
* src/wget.c: Add struct _body_callback_context,
  always call bar_slot_begin() on begin of new download,
  use downloader->id as progress slot
* src/wget_bar.h: Amend prototypes of bar_ functions
2016-10-13 16:39:59 +02:00
968e665973 Call bar_update_slot() from main thread
* libwget/bar.c: Fix typos,
  (wget_bar_set_slots): Init mem only when needed,
  add comments describing Escape sequences
* src/bar.c (bar_update_slots): Take nslots as param
* src/wget.c: Move call to bar_update_slots() from worker to main thread
* src/wget_bar.h: Amend prototype of bar_update_slots()
2016-10-12 17:09:09 +02:00
753c95473e * docs/wget2_manual.md: More changes for Wget2 2016-10-12 16:14:16 +02:00
e068771723 * src/wget.c: Fix quota calculation 2016-10-11 17:03:33 +02:00
21099bcf7f * libwget/http.c: Fix update of downloaded bytes for progress bar 2016-10-11 15:56:54 +02:00
b908178f53 * libwget/ssl_gnutls.c: Print debug messages using debug_printf() 2016-10-11 15:55:35 +02:00
fb14f1e412 * src/options.c: Amend Wget2 banner string 2016-10-11 14:42:32 +02:00
f1aacb5bcf * src/host.c (host_queue_free): Also free robots.txt job 2016-10-11 12:35:39 +02:00
8f45be7d2b * configure.ac: Ignore the Stack Protector warnings 2016-10-10 15:47:11 +02:00
bc0fc2b28d * libwget/http.c: Support compression for HTTP/2 2016-10-10 15:44:55 +02:00
2da0ad63dd Improved testing on Travis
* .travis.sh: Rewrite for clarity and speed. Testing shows that valgrind
and ASan don't play well together. So split their executions.
* .travis.yml: Enable ccache for builds. Also try to update libc6 if
possible
* .travis.yml: Do not run on OSX with CC=gcc. This is because on OSX,
gcc is only a wrapper around clang.
* .travis_setup.sh: Cosmetic changes only
* Makefile.am (check-coverage): Explcicitly disable valgrind tests when
checking coverage
* tests/libtest.c (wget_test): Use a provided suppressions file for
valgrind. This is required since the valgrind version on Travis seems to
be out of sync with libc. This causes valgrind to report a memory bug
falsely.
* tests/valgrind-suppressions: New file with some valgrind suppressions.
Included suppressions are for a valgrind/libc bug and to suppress
reachable memory messages from within gnutls
2016-10-07 17:51:48 +02:00
86f154012f Revert "Test travis"
This reverts commit 75daf2cdc7.
I screwed up. Pushed to the wrong repository
2016-10-06 18:14:49 +02:00
75daf2cdc7 Test travis 2016-10-06 18:13:13 +02:00
bf9609f27f * src/wget.c (nop): Add ifddef guards around SIGWINCH 2016-10-05 12:31:53 +02:00
49ab9740e1 Fix progress bar details
* include/wget/wget.h: wget_get_screen_size() must not be pure
* libwget/bar.c (_bar_get_width): Leave one space right to avoid linebreaks
* libwget/utils.c (wget_human_readable): Align # of bytes
* src/bar.c (wget_get_screen_size): Fix return value
2016-10-05 11:49:45 +02:00
40340af300 Export a new function for determining screen dimensions
* include/wget/wget.h: New function wget_get_screen_size. Replaces old
wget_determine_screen_width
* libwget/bar.c (_bar_get_width): Use wget_get_screen_size instead of
wget_determine_screen_width
* libwget/utils.c: Replace function wget_determine_screen_width with a
more generic wget_get_screen_size.
2016-10-04 21:31:34 +02:00
354c8624d1 Handle SIGWINCH for progress bar
* include/wget/wget.h: New function wget_bar_screen_resized, as an
indicator that the screen size may have changed. Also, wget_bar_update
may now modify the *bar parameter
* libwget/bar.c: Move detection code for screen size into a separate
function called _bar_get_width.
  (wget_bar_update): Check if the screen width has changed and
reallocate more space if necessary
* src/wget.c: Install a new signal handler for SIGWINCH
2016-10-04 21:30:00 +02:00
e10cf1d1f3 Move window size detection to libwget
* include/wget/wget.h: Declare new function wget_determine_screen_width.
Also change prototype of wget_bar_init to not accept a max_width
parameter
* src/utils.c: Move function determine_screen_width from here ...
* libwget/utils.c: ... to here.
* src/utils.{c,h}: Delete empty file
* src/Makefile.am: Remove source files utils.{c,h}
* src/bar.c: Move detection of screen width from here ...
* libwget/bar.c: ... to here
* tests/test.c: wget_bar_init() no longer takes a max_width parameter
2016-10-04 20:29:30 +02:00
4cdffd2def Cleaner API for progress bar within libwget
* include/wget/wget.h (_wget_bar_ctx): Remove unused elements
expected_size and filename. Rename slotpos to _slotpos to reflect that
it should not be used by the client.
* include/wget/wget.h: Remove declaration for old API functions
bar_register and bar_deregister. Define declarations for
wget_bar_slot_begin, wget_bar_slot_register and
wget_bar_slot_deregister.
* libwget/bar.c (_bar_slot_status_t): Define new enum to store the
status of a slot of the progress bar.
  (_bar_slot_t): Remove unused element last_ctx. Also, define slot
  elements: filename, file_size, bytes_downloaded and status.
  (wget_bar_slot_begin): Define new function. Invoked when the client
  has started downloading a new file.
  (wget_bar_slot_register): New function to register a new file in a
  progress bar slot.
  (wget_bar_deregister): Remove old function
  (wget_bar_slot_deregister): New function, called after file has
  finished downloading. Successfully or unsuccessfully.
  (wget_bar_deregister): Remove old function
  (_bar_set_progress): last_ctx is no longer used. Instead data is
  stored within the slot struct itself
  (_bar_update_slot): Same
  (_bar_print_final): Same
  (wget_bar_deinit): Free the filename stored in the slot
* src/bar.c: Edit to reflect the changes in libwget API
* src/wget.c (_get_header): Same
  (http_send_request): Same
  (http_receive_response): Same
* src/wget_bar.h: Same
2016-10-04 20:23:56 +02:00
6a115f1b1e * libwget/bar.c: Remove mutex stdout
The mutex stdout was meant to ensure that multiple routines don't write
to the screen concurrently. However, this mutex is not required, since
such syncronization should be handled by the client using the library.
All other library methods for the progress avoid any form
synchronization primitives for this reason
2016-10-03 13:12:31 +02:00
01706e90c5 * src/bar.c: Rename mutex to bar_mutex 2016-10-03 13:08:46 +02:00
7be16cf925 * libwget/http.c: Remove unused struct _body_callback_context 2016-10-03 13:04:22 +02:00
554a34422b Better code coverage for HTML parsing
* tests/test-idn-meta.c: Add HTML5 meta charset
* tests/test-meta-robots.c: Add meta robots 'all' and 'follow'
2016-09-30 16:30:32 +02:00
165cfa20fb Mark some functions as pure
* libwget/cookie.c (_compare_cookie{,2}): Mark as pure
* libwget/encoding.c (_utf8_is_valid): Same
* libwget/hsts.c (_compare_hsts): Same
* libwget/ocsp.c (_compare_ocsp): Same
* libwget/netrc.c (_compare_netrc): Same
* libwget/tls_session.c (_compare_tls_session): Same
2016-09-30 14:48:08 +02:00
9568b0c87e Rename include/libwget.h to include/wget/wget.h+wgetver.h 2016-09-30 09:47:32 +02:00
a9c509898b Add tests for CSS parsing
* tests/Makefile.am: Add test-parse-css.c
* tests/test-parse-css.c: New file including two tests
2016-09-29 16:23:32 +02:00
c2ffbb889c * Makefile.am: Remove unwanted stuff from coverage report 2016-09-29 16:23:09 +02:00
21c2bb7339 Reduce Travis OSX build overhead 2016-09-29 12:27:35 +02:00
4ac70bf7c3 * tests/test.c: Add wget_bar_* unit test 2016-09-29 11:53:04 +02:00
51c1c42e7a * libwget/bar.c: Cosmetics, param check for wget_bar_init() 2016-09-29 11:52:23 +02:00
a8d6956247 Don't dereference NULL pointer when copying filename
* libwget/bar.c (wget_bar_deregister): Make sure filename is not NULL
before calling strdup() on it
2016-09-29 09:33:38 +02:00
849cc8acd0 * libwget/http.c: Check WITH_LIBNGHTTP2 before referring nghttp2_nv 2016-09-28 19:36:30 +02:00
664cbc14ee Fix race condition when pipelining reuqests
* libwget/http.c: Add streamid to debug messages
* src/progress.h: Remove member downloader and head from _body_callback_context
* src/wget.c (downloader_thread): Keep track of locking,
  (_get_header): Remove reference to downloader,
  (http_send_request): Replace downloader with job

With HTTP/2 pipelining (parallel streams per connection) is active.
The bug fixed stops file descriptor leaks and writing data into wrong
files.
2016-09-28 17:07:19 +02:00
aa442c14bf * README.md: Add build status for OpenCSW 2016-09-28 11:07:09 +02:00
a13166b0b9 * src/net.c (wget_tcp_resolve): Fix mutex unlocking 2016-09-28 10:36:24 +02:00
f5d3aae7b9 * src/wget.c (add_url, process_response): Fix NULL pointer dereference
Reported-by: Coverity
2016-09-28 10:35:14 +02:00
ce828e64d3 * src/wget.c (establish_coneection): Don't crash when no mirrors defined for metalink file 2016-09-27 22:24:33 +02:00