39dfeec1ae
* bootstrap.conf: Add canonicalize module.
* cfg.mk: Added exclusions to no new line at EOF for binary files used in tests.
* configure.ac: Check for libgpgme
* docs/wget2_manual.md: Added manual entries for --verify-sig, --gnupg-homedir, and new exit codes.
* fuzz/Makefile.am: Amend linker options
* include/wget/wget.h: Add error codes for GPG signature verification.
* po/POTFILES.in: Add new files.
* src/Makefile.am: Add src/gpgme.c and src/wget_gpgme.h
* src/gpgme.c: Using new error codes, and better signature verification failure detection.
* src/job.c (job_free): Free sig_filename
* src/options.c: Add new options --gnupg-homedir and --verify-sig
* src/wget.c: Verify the signature if possible, exit with the proper status code,
print error messages
* src/wget_gpgme.h: Add documentation.
* src/wget_job.h: Extend struct JOB
* src/wget_options.h: Add new status codes, add new members to struct config
* tests/Makefile.am: Added new tests.
* tests/gpg-test-util.h: Common GPG testing functionality (tries to verify a signature, expects exit code ... etc).
* tests/gpg/helloworld.txt: Text that has been signed in the .sig files.
* tests/gpg/helloworld.txt.{invalid,no-pub,missing,trusted}.sig: Signatures on helloworld.txt with properties described by their names.
* tests/gpg/openpgp-revocs.d/*: Revocation certs for the fake "homedir"
* tests/gpg/private-keys-v1.d/*: The not-so-private private keys used to sign the test file.
* tests/gpg/pubring.kbx: Fake gnupg homedir public key ring.
* tests/gpg/test-gpg-*.c: Tests.
* tests/valgrind-supressions: Supress failures due to oddness with GPGME.
* unit-tests/Makefile.am: Add src/gpgme.o to BASE_OBJS
Fuzzers
These are fuzzers designed for use with libFuzzer or afl. They can
be used to run on Google's OSS-Fuzz (https://github.com/google/oss-fuzz/).
The convention used here is that the initial values for each parser fuzzer are taken from the $NAME.in directory.
Crash reproducers from OSS-Fuzz are put into $NAME.repro directory for regression testing with top dir 'make check' or 'make check-valgrind'.
Running a fuzzer using clang
Use the following commands on top dir:
export CC=clang-6.0
export CFLAGS="-O1 -g -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=undefined,integer,nullability -fsanitize=address -fsanitize-address-use-after-scope -fsanitize-coverage=trace-pc-guard,trace-cmp"
export LIB_FUZZING_ENGINE="-lFuzzer -lstdc++"
./configure --enable-fuzzing --disable-doc --disable-manywarnings
make clean
make -j$(nproc)
cd fuzz
# run libwget_xml_parse_buffer_fuzzer
UBSAN_OPTIONS=print_stacktrace=1 ASAN_SYMBOLIZER_PATH=/usr/lib/llvm-6.0/bin/llvm-symbolizer \
./run-clang.sh libwget_xml_parse_buffer_fuzzer
If you see a crash, then a crash corpora is written that can be used for further investigation. E.g.
==2410==ERROR: AddressSanitizer: heap-use-after-free on address 0x602000004e90 at pc 0x00000049cf9c bp 0x7fffb5543f70 sp 0x7fffb5543720
...
Test unit written to ./crash-adc83b19e793491b1c6ea0fd8b46cd9f32e592fc
To reproduce the crash:
./libwget_xml_parse_buffer_fuzzer < ./crash-adc83b19e793491b1c6ea0fd8b46cd9f32e592fc
You can also copy/move that file into libwget_xml_parse_buffer_fuzzer.repro/
and re-build the project without fuzzing for a valgrind run, if you like that better.
Just a ./configure (maybe with sanitizers enabled) and a make check-valgrind should reproduce it.
Running a fuzzer using AFL
Use the following commands on top dir:
$ CC=afl-clang-fast ./configure --disable-doc
$ make -j$(nproc) clean all
$ cd fuzz
$ ./run-afl.sh libwget_xml_parse_buffer_fuzzer
Fuzz code coverage using the corpus directories *.in/
Code coverage reports currently work best with gcc+lcov+genhtml.
In the top directory:
CC=gcc CFLAGS="-O0 -g" ./configure --disable-doc --disable-manywarnings
make fuzz-coverage
xdg-open lcov/index.html
Each fuzzer target has it's own files/functions to cover, e.g.
libwget_xml_parse_buffer covers libwget/xml.c (except
wget_xml_parse_file() and wget_html_parse_file()).
To work on corpora for better coverage, cd fuzz and use e.g.
./view-coverage.sh libwget_xml_parse_buffer_fuzzer.
Creating wget_options_fuzzer.dict
for i in `../src/wget2 --help|tr ' ' '\n'|grep ^--|cut -c 3-|sort`;do echo \"$i\"; done >wget_options_fuzzer.dict