73fe1098d2
Switch to C11 threads
...
This change also removes all wget_thread_* functionality from libwget.
Since gnulib also switched to relying on C11, there is no way we can stay
with a lower C standard. This is how it is.
2022-08-14 14:53:09 +02:00
8892867db0
* examples/getstream.c: Initialize array to avoid strduping random memory
2022-06-26 13:03:50 +02:00
44f1bb7c4c
Remove memset() calls
...
* examples/batch_loader.c (main): Likewise
* examples/check_url_types.c (main): Likewise
* libwget/metalink.c (add_file_hash): Likewise
* src/wget.c (program_init): Likewise
2022-06-25 13:57:00 +02:00
b22e6b8b07
* examples/Makefile.am: Remove linking of libgnu.la
2022-06-25 13:38:31 +02:00
3dc7f71098
Update copyright year
2022-02-25 17:46:43 +01:00
214a6f4937
Use sa_handler instead of sa_sigaction
...
* examples/batch_loader.c (main): Use sa_handler instead of sa_sigaction.
* examples/check_url_types.c (main): Likewise.
* src/wget.c (program_init): Likewise.
2021-12-05 17:28:51 +01:00
22162f82b2
Update copyright year
2021-01-22 21:58:38 +01:00
b3fb93ea46
Fix findings from scan-build 11
...
* examples/batch_loader.c (_normalize_location): 'rc' was set but not used.
* examples/check_url_types.c (_normalize_location): 'rc' was set but not used.
2020-12-20 17:13:05 +01:00
ef40fb5221
Parse HTML download attribute
...
* examples/print_html_urls.c: Print download attribute.
* include/wget/wget.h (wget_html_parsed_url): Add download member.
* libwget/html_url.c (html_get_url): Parse download attribute.
2020-03-15 11:53:09 +01:00
59d9ecd9c0
* Update copyright year to 2020
2020-01-10 00:33:02 +01:00
0f624b16b5
Update Makefile.am for latest gnulib
...
* examples/Makefile.am: Amend LDADD assignment
* fuzz/Makefile.am: Likewise
* libwget/Makefile.am: Likewise
* src/Makefile.am: Likewise
* tests/Makefile.am: Likewise
* unit-tests/Makefile.am: Likwise
2020-01-07 15:46:12 +01:00
4b9edbad2d
Update Copyright statements to be compatible with update-copyright module
2019-09-10 23:41:56 +02:00
d4af0fa1f6
* examples/http_get.c: Use https:// instead of http://
2019-09-06 00:30:08 +02:00
3ce5137d5b
Rename wget_html_parsed_url.dir -> wget_html_parsed_url.tag
...
* include/wget/wget.h: Rename wget_html_parsed_url.dir -> wget_html_parsed_url.tag
* examples/print_html_urls.c: Likewise
* libwget/html_url.c: Likewise
* src/wget.c: Likewise
2019-09-02 17:37:32 +02:00
a6e38ee2ec
* examples/Makefile.am: Add -DMALLOC_RETURNS_NONNULL to AM_CFLAGS and AM_CPPFLAGS
2019-08-17 18:34:45 +02:00
987705cd7e
* examples/batch_loader.c: Check return value of wget_strmemdup()
2019-08-16 17:22:32 +02:00
dd5a7d81c9
* examples/check_url_types.c: Check return value of wget_strmemdup()
2019-08-16 17:22:13 +02:00
65ec4901b4
Remove global wget_iri_schemes[], add wget_iri_scheme enum
...
* include/wget/wget.h: Remove global wget_iri_schemes[],
add wget_iri_scheme enum
* libwget/iri.c: Add struct iri_scheme,
remove wget_iri_schemes and iri_ports,
new function wget_iri_scheme_get_name(),
fix code
* examples/check_url_types.c: Use comparison instead of wget_strcasecmp
* fuzz/libwget_iri_fuzzer.c: Use WGET_IRI_SCHEME_HTTPS instead of string
* libwget/http.c: Use wget_iri_scheme_get_name()
* libwget/http.h: Change scheme from string to wget_iri_scheme
* libwget/http_parse.c: Fix wget_http_get_scheme()
* src/blacklist.c: Fix hash_iri()
* src/host.c: Fix _host_hash()
* src/options.c: Use WGET_IRI_SCHEME_* instead of string
* src/stats_server.c: Use wget_iri_scheme for scheme member
* src/wget.c: Fix code
* src/wget_host.h: Use wget_iri_scheme fro scheme member
* unit-tests/test.c: Fix tests
2019-08-13 16:55:53 +02:00
3835d892bb
Rename HTTP parse callback types
...
* include/wget/wget.h: Rename HTTP parse callback types
* examples/getstream.c: Likewise
* libwget/http.c: Likewise
* libwget/http_highlevel.c: Likewise
2019-08-12 11:38:10 +02:00
85a7ee75a4
examples/http_get2.c: Cleanup
2019-08-12 10:50:21 +02:00
12f1a03693
Changed G_GNUC_WGET prefix to WGET_GCC
...
* include/wget/wget.h: Changed G_GNUC_WGET prefix to WGET_GCC
* */*.[ch]: Likewise
2019-08-08 17:13:24 +02:00
837ab09eb5
Add examples/relative_to_absolute_url.c
2019-08-08 15:49:51 +02:00
37ce0d0a15
* examples/getstream.c: Fix call to wget_http_get()
2019-07-25 11:58:12 +02:00
e6aba6bb09
Rename wget_http_connection_t -> wget_http_connection
...
* include/wget/wget.h: Rename wget_http_connection_t -> wget_http_connection
* examples/check_url_types.c: Likewise
* examples/http_get.c: Likewise
* examples/http_get2.c: Likewise
* examples/http_multi_get.c: Likewise
* examples/websequencediagram.c: Likewise
* examples/websequencediagram_high.c: Likewise
* fuzz/libwget_http_client_fuzzer.c: Likewise
* libwget/http.c: Likewise
* libwget/http_highlevel.c: Likewise
* libwget/http_parse.c: Likewise
* libwget/ssl_gnutls.c: Likewise
* src/wget.c: Likewise
* src/wget_job.h: Likewise
2019-07-19 16:47:01 +02:00
35c18afa13
Rename wget_http_request_t -> wget_http_request
...
* include/wget/wget.h: Rename wget_http_request_t -> wget_http_request
* examples/http_get2.c: Likewise
* examples/http_multi_get.c: Likewise
* examples/websequencediagram.c: Likewise
* fuzz/libwget_http_client_fuzzer.c: Likewise
* libwget/http.c: Likewise
* libwget/http_highlevel.c: Likewise
* libwget/http_parse.c: Likewise
* libwget/ssl_gnutls.c: Likewise
* src/wget.c: Likewise
* unit-tests/test.c: Likewise
2019-07-19 16:44:21 +02:00
da75b622d6
Rename wget_http_response_t -> wget_http_response
...
* include/wget/wget.h: Rename wget_http_response_t -> wget_http_response
* examples/batch_loader.c: Likewise
* examples/check_url_types.c: Likewise
* examples/getstream.c: Likewise
* examples/http_get.c: Likewise
* examples/http_get2.c: Likewise
* examples/http_multi_get.c: Likewise
* examples/websequencediagram.c: Likewise
* examples/websequencediagram_high.c: Likewise
* fuzz/libwget_http_client_fuzzer.c: Likewise
* fuzz/libwget_http_parse_fuzzer.c: Likewise
* libwget/http.c: Likewise
* libwget/http_highlevel.c: Likewise
* libwget/http_parse.c: Likewise
* libwget/ssl_gnutls.c: Likewise
* src/gpgme.c: Likewise
* src/stats_site.c: Likewise
* src/wget.c: Likewise
* src/wget_gpgme.h: Likewise
* src/wget_stats.h: Likewise
* tests/test-cookies-http_state.c: Likewise
* unit-tests/test-cookies-http_state.c: Likewise
* unit-tests/test.c: Likewise
2019-07-19 16:41:21 +02:00
8c47901ef8
Rename wget_html_parsed_result_t -> wget_html_parsed_result
...
* include/wget/wget.h: Rename wget_html_parsed_result_t -> wget_html_parsed_result
* examples/check_url_types.c: Likewise
* examples/print_html_urls.c: Likewise
* fuzz/libwget_html_url_fuzzer.c: Likewise
* libwget/html_url.c: Likewise
* src/wget.c: Likewise
2019-07-19 16:14:33 +02:00
74928eaf59
Rename wget_html_parsed_url_t -> wget_html_parsed_url
...
* include/wget/wget.h: Rename wget_html_parsed_url_t -> wget_html_parsed_url
* examples/check_url_types.c: Likewise
* examples/print_html_urls.c: Likewise
* libwget/html_url.c: Likewise
* src/wget.c: Likewise
2019-07-19 16:13:07 +02:00
eb7e935d22
Rename wget_string_t -> wget_string
...
* include/wget/wget.h: Rename wget_string_t -> wget_string
* examples/batch_loader.c: Likewise
* examples/check_url_types.c: Likewise
* examples/print_html_urls.c: Likewise
* libwget/atom_url.c: Likewise
* libwget/robots.c: Likewise
* libwget/rss_url.c: Likewise
* libwget/sitemap_url.c: Likewise
* src/host.c: Likewise
* src/wget.c: Likewise
* unit-tests/test.c: Likewise
2019-07-19 16:10:54 +02:00
29bbd6185d
Rename wget_css_parsed_url_t -> wget_css_parsed_url
...
* include/wget/wget.h: Rename wget_css_parsed_url_t -> wget_css_parsed_url
* examples/print_css_urls3.c: Likewise
* libwget/css_url.c: Likewise
2019-07-19 16:07:41 +02:00
e61d3298e5
Rename wget_ocsp_db_t -> wget_ocsp_db
...
* include/wget/wget.h: Rename wget_ocsp_db_t -> wget_ocsp_db
* examples/http_multi_get.c: Likewise
* fuzz/libwget_ocsp_fuzzer.c: Likewise
* libwget/ocsp.c: Likewise
* libwget/plugin.c: Likewise
* libwget/ssl_gnutls.c: Likewise
* libwget/ssl_wolfssl.c: Likewise
* src/plugin.c: Likewise
* src/wget_options.h: Likewise
* src/wget_plugin.h: Likewise
* tests/test-plugin-dummy.c: Likewise
2019-07-19 15:59:58 +02:00
618f49160d
Rename wget_cookie_db_t -> wget_cookie_db
...
* include/wget/wget.h: Rename wget_cookie_db_t -> wget_cookie_db
* examples/http_get2.c: Likewise
* fuzz/libwget_cookie_fuzzer.c: Likewise
* libwget/cookie.c: Likewise
* libwget/http_highlevel.c: Likewise
* libwget/init.c: Likewise
* src/wget_options.h: Likewise
* tests/test-cookies-http_state.c: Likewise
* unit-tests/test-cookies-http_state.c: Likewise
* unit-tests/test.c: Likewise
2019-07-18 13:11:55 +02:00
50be5af992
Rename wget_iri_t -> wget_iri
...
* include/wget/wget.h: Rename wget_iri_t -> wget_iri
* examples/*.c: Likewise
* fuzz/*.c: Likewise
* libwget/*.c: Likewise
* src/*.c: Likewise
* tests/*.c: Likewise
* unit-tests/*.c: Likewise
2019-07-18 13:10:27 +02:00
36bcc83b9a
Rename wget_thread_t -> wget_thread
...
* include/wget/wget.h: Rename wget_thread_t -> wget_thread
* examples/batch_loader.c: Likewise
* libwget/thread.c: Likewise
* src/*.c: Likewise
* unit-tests/test-cond.c: Likewise
2019-07-18 12:50:41 +02:00
bdbcbb7992
Rename wget_vector_t -> wget_vector
...
* include/wget/wget.h: Rename wget_vector_t -> wget_vector
* examples/print_css_urls3.c: Likewise
* fuzz/*.c: Likewise
* libwget/*.c: Likewise
* src/*.c: Likewise
* tests/*.c: Likewise
* unit-tests/*.c: Likewise
2019-07-18 12:35:37 +02:00
db50c6a801
Rename wget_buffer_t -> wget_buffer
...
* include/wget/wget.h: Rename wget_buffer_t -> wget_buffer
* examples/*.c: Likewise
* fuzz/libwget_iri_fuzzer.c: Likewise
* libwget/*.c: Likewise
* src/*.c: Likewise
* tests/*.c: Likewise
* unit-tests/*.c: Likewise
2019-07-18 12:30:57 +02:00
baa5ef562f
Add sc_prohibit_free to 'make syntax-check'
...
* cfg.mk: Add sc_prohibit_free rule plus exceptions
* examples/print_css_urls3.c: Remove unneeded comment
* fuzz/libwget_base64_fuzzer.c: Use wget_free() instead of free()
* fuzz/libwget_cookie_fuzzer.c: Likewise
* fuzz/libwget_css_url_fuzzer.c: Likewise
* fuzz/libwget_http_client_fuzzer.c: Likewise
* fuzz/libwget_utils_fuzzer.c: Likewise
* libwget/vector.c: Likewise
* src/options.c: Likewise
* src/wget_main.h: Likewise
* tests/test-compression.c: Likewise
* tests/test-cookies-http_state.c: Likewise
* tests/test-metalink.c: Likewise
* unit-tests/stringmap_perf.c: Likewise
This rule checks for calling free(). Instead, either use wget_free() or
the convenience macro xfree().
2019-06-24 16:04:23 +02:00
b6c2f5f75c
Put resolver/dns code into own source file
...
* docs/Makefile.am: Add libwget-dns.3 man page
* examples/batch_loader.c: Remove call to wget_tcp_set_dns_timeout()
* examples/check_url_types.c: Use wget_dns_set_timeout()
* include/wget/wget.h: Add wget_dns_ API
* libwget/Makefile.am: Add dns.c
* libwget/dns.c: New file
* libwget/init.c: Use wget_dns_set_caching()
* libwget/net.c: Remove DNS functions
* libwget/net.h (wget_tcp_st): Add dns member
* po/POTFILES.in: Add libwget/dns.c
* src/options.c: Use new DNS API
* src/stats.c: Add some checks
* src/stats_dns.c: Use wget_dns_get_stats()
2019-06-21 13:10:47 +02:00
cdb3600791
Update copyrigght to 2019
2019-01-02 12:42:53 +01:00
f04b8eef66
Add example batch_loader
...
* examples/Makefile.am: Add batch_loader.c
* examples/batch_loader.c: New example code
This example uses 500 threads to process domains / URLS from
stdin. It downloads the URL and creates a stats entry in out.csv.
Check the source for details.
2018-08-04 21:56:55 +02:00
3688ffb941
Update copyright to 2018
2018-04-30 20:52:11 +02:00
863338764a
Dont't recognize %3A as colon in relative URLs
...
* include/wget/wget.h: New function wget_iri_unescape_url_inline()
* libwget/iri.c: Likewise
* src/wget.c (_normalize_uri): Use wget_iri_unescape_url_inline()
* examples/check_url_types.c: Likewise
* tests/test-base.c: Add test case
* unit-tests/test.c (test_iri_relative_to_absolute): Add test case
Relative URLS like in href='foo%3A/' have been detected and parsed
as absolute URL with (unknown) scheme 'foo:'.
This commit fixes it.
2018-04-13 19:23:11 +02:00
f44de7d125
Rename WGET_HTML_PARSED_RESULT to wget_html_parsed_result_t
...
* examples/check_url_types.c: Likewise
* examples/print_html_urls.c: Likewise
* fuzz/libwget_html_url_fuzzer.c: Likewise
* include/wget/wget.h: Likewise
* libwget/html_url.c: Likewise
* src/wget.c: Likewise
2017-10-22 21:05:15 +02:00
8fb49619b7
Rename WGET_HTML_PARSED_URL to wget_html_parsed_url_t
...
* examples/check_url_types.c: Likewise
* examples/print_html_urls.c: Likewise
* include/wget/wget.h: Likewise
* libwget/html_url.c: Likewise
* src/wget.c: Likewise
2017-10-22 21:03:31 +02:00
bbf7429040
Rename WGET_PARSED_URL to wget_css_parsed_url_t
...
* examples/print_css_urls3.c: Likewise
* include/wget/wget.h: Likewise
* libwget/css_url.c: Likewise
2017-10-22 21:02:14 +02:00
29e241cad7
Fix usage of wget_global_init() and wget_http_get()
...
* examples/*.c: Use 0 instead of NULL as terminating param
* fuzz/main.c: Likewise
* include/wget/wget.h: Remove G_GNUC_WGET_NULL_TERMINATED from
wget_global_init() and wget_http_get().
2017-10-09 16:56:53 +02:00
4c02e13630
Reduce variable scope
...
* examples/print_html_urls.c: Likewise
* fuzz/libwget_cookie_fuzzer.c: Likewise
* fuzz/main.c: Likewise
* tests/libtest.c: Likewise
* unit-tests/test.c: Likewise
Found by: cppcheck
2017-10-07 13:29:09 +02:00
98c8e410a8
Add plugin APIs for custom HSTS, HPKP and OCSP databases
...
* include/wget/wget.h: Add plugin API declarations
* libwget/hsts.c: Make wget_hsts_db_t extensible
* libwget/hpkp.c: Make wget_hpkp_db_t extensible
* libwget/ocsp.c: Make wget_ocsp_db_t extensible
* libwget/plugin.c: Add virtual functions for plugin API
* src/plugin.c: Add plugin API implementation
* src/wget_plugin.h: Likewise
* src/option.c: Add code to call the implementation
* src/wget.c: Likewise
* tests/test-plugin-dummy: Add plugin with dummy databases for testing
* tests/Makefile.am: Add target libplugindb.la
* tests/test-plugin.c: Add tests for custom databases
* unit-tests/test.c: Add tests for new HPKP functions
* libwget/ssl_gnutls.c: Adapt to API changes
* examples/http_multi_get.c: Likewise
* fuzz/libwget_hpkp_fuzzer.c: Likewise
* fuzz/libwget_hsts_fuzzer.c: Likewise
* fuzz/libwget_ocsp_fuzzer.c: Likewise
This feature implemented as per the specifications at
https://gitlab.com/akash_rawal/wget2/wikis/Stage-4
2017-09-03 18:21:39 +02:00
c1ce6dec27
Remove h2-14 and h2-16 from ALPN
...
* libwget/ssl_gnutls.c: Remove h2-14 and h2-16 from default ALPN offer
* examples/http_multi_get.c: Set ALPN to h2,http/1.1
* src/options.c: Likewise
h2-14 and h2-16 were some HTTP/2 drafts, should not be relevant in
the real world now.
2017-07-04 09:24:35 +02:00
d9af6d266f
Open files in binary mode where needed
...
* examples/websequencediagram.c: Use binary mode on file open
* libwget/css.c: Likewise
* libwget/hashfile.c: Likewise
* libwget/http_highlevel.c: Likewise
* libwget/io.c: Likewise
* libwget/xml.c: Likewise
* src/job.c: Likewise
* src/options.c: Likewise
* src/wget.c: Likewise
* tests/libtest.c: Likewise
* tests/stringmap_perf.c: Likewise
This fixes discrepancies between stat/st_size and read bytes on
MinGW / Wine / Windows.
2017-05-20 22:03:45 +02:00
