* include/wget/wget.h: Convert wget_http_connection_t to forward declaration
* libwget/http.h: Move wget_http_connection_t definition here
* libwget/Makefile.am: Add new file http.h
* libwget/http.c: Include http.h and add accessor functions
* libwget/http_highlevel.c: Include http.h
* src/wget.c: Replace access to wget_http_connection_t with accessor function calls
* include/wget/wget.h: Rename and remove gnulib deps.
* libwget/strlcpy.c: Rename function and fall back to strlcpy.
* */*.c: Rename strlcpy to wget_strlcpy.
* tests/test.c (test_set_proxy): Add tests for wget_http_set_http_proxy()
* tests/test.c (test_set_proxy): Add tests for wget_http_set_https_proxy()
* libwget/http.c (wget_http_set_http_proxy): Return length of http_proxies if no parse error
* libwget/http.c (wget_http_set_https_proxy): Return length of https_proxies if no parse error
* include/wget/wget.h: New functions wget_decompress_set_error_handler()
and wget_decompress_get_context().
* libwget/decompressor.c: New functions wget_decompress_set_error_handler()
and wget_decompress_get_context().
* libwget/http.c: Implement and set decompressor error handler.
* include/wget/wget.h: Add flag 'proxied' to wget_http_connection_t,
add arg 'proxied' to wget_http_request_to_buffer(),
add new function declaration wget_http_set_no_proxy(),
add new function declaration wget_http_match_no_proxy().
* libwget/http.c: Add variable 'no_proxies',
(wget_http_open): Check host against 'no_proxy' and set 'conn->proxied'
when appropriate,
(wget_http_send_request): Call wget_http_request_to_buffer() with
conn->proxied,
(wget_http_request_to_buffer): Check 'proxied',
(_parse_proxies): Cleanup,
(_parse_no_proxies): New function,
(wget_http_set_no_proxy): New function,
(wget_http_match_no_proxy): New function
* libwget/init.c (wget_global_deinit): Free no_proxy memory
* libwget/ip.c: Add code in comments (for later use)
* libwget/utils.c (wget_match_tail, wget_match_tail_nocase): Cleanup
* src/options.c (init): Handle 'no_proxy' env variable
* src/wget_options.h (struct config): Add no_proxy member
* libwget/cookie.c: Use atoll() instead of atol()
* libwget/hsts.c: Likewise
* libwget/http.c: Likewise
* libwget/ocsp.c: Likewise
* libwget/tls_session.c: Likewise
* libwget/hpkp.c: Change type of maxage from long to time_t
* libwget/http.c (wget_http_get_response_cb): Save data when chunk
complete in one data packet.
* tests/Makefile.am: Add new test 'test-chunked'
* tests/test-chunked.c: New test
* configure.ac: Suppress -Wunsuffixed-float-constants for gcc,
add -Wno-disabled-macro-expansion to clang flags
* examples/getstream.c: Removed 'unused' attributes from argc and argv.
* include/wget/wget.h: Fix slot to int for bar functions.
* libwget/bar.c: Fix slot to int for bar functions, cast to int,
change type of bitfield 'redraw' to unsigned.
* libwget/hsts.c (wget_hsts_new): Fix int to bitfield conversion.
* libwget/http.c: Cast printf %p format to (void *).
* src/wget.c: Cast printf %p format to (void *),
remove ACTION_DONE from enum actions.
* src/wget_host.h: Fix double declaration of host_remove_job()
* tests/libtest.c (wget_test_check_filesystem): Fix type of 'rc'.
* tests/test-idn-robots.c: Fix comma to semikolon at end of line.
* tests/test.c (test_strcasecmp_ascii): Fix type of loop variable.
* libwget/http.c (wget_http_create_request): Add 'Host' header
to header list,
(wget_http_send_request): Do not add 'Host' hard-coded,
(wget_http_request_to_buffer): Special treatment of 'Content-Length'
* src/wget.c (http_create_request): Replace wget's HTTP headers by
user-provided headers, except 'Cookie' headers which will be appended.
* libwget/http.c: Add streamid to debug messages
* src/progress.h: Remove member downloader and head from _body_callback_context
* src/wget.c (downloader_thread): Keep track of locking,
(_get_header): Remove reference to downloader,
(http_send_request): Replace downloader with job
With HTTP/2 pipelining (parallel streams per connection) is active.
The bug fixed stops file descriptor leaks and writing data into wrong
files.
Give the progress bar its own thread and let it update the entire
display asynchronously at a specified time interval. The existing
implementation refreshes the progress bar for each network packet
downloaded. Over multiple downloader threads and a high speed network
connection this can lead to far too many redrawings of the screen. Also,
each of the downloader threads will block while trying to acquire the
thread mutex because another thread just retrieved a packet. While I
haven't profiled it, it seems like there would be extremely high lock
contention in the existing implementation. Instead, with a separate
thread, we can update all the progress bar slots simultaneously at
regular intervals.
* bootstrap.conf: Include Gnulib module "ioctl"
* include/libwget.h.in: Define always_inline, flatten and deprecated
compiler attributes for both GCC and Clang
* include/libwget.h.in: Export new functions wget_human_readable,
wget_bar_register and wget_bar_deregister.
* include/libwget.h.in (wget_http_response_t): Add new element
"cur_downloaded" to struct. This element keeps a track of the raw
number of bytes downloaded in the response
(_wget_bar_ctx): Define new struct for storing the progress bar
context.
* libwget/bar.c: Fix display of downloaded ratio. Ensure it does not
exceed 100% by comparing the raw downloaded bytes instead of
uncompressed downloaded bytes.
Add support for printing the filename and downloaded bytes to the
progress bar.
* libwget/http.c (wget_http_response_cb): Store the raw number of
bytes downloaded from the network in the response data
* libwget/utils.c (wget_human_readable): New function to convert an
int value to a human readable string
* src/bar.c: Use the entire screen width instead of just 70 columns
for the progress bar.
Provide methods to register and deregister a progress bar
context
(wget_bar_update_thread): New method for the progress bar thread
to update the output periodically.
* src/bar.h: Same
* src/job.h: Move definition of struct DOWNLOADER from main.c to
here
* src/progress.h: Move definition of _body_callback_context from
main.c to here
* src/utils.c: Add new method to determine the width of the screen
* src/utils.h: Same
* src/wget.c: Use the new progress bar contexts. Update, register
and deregister them
(main): Deinit the progress bar only if it was enabled
* src/Makefile.am: Add new files, progress.h, utils.c, utils.h
