This commit is a large squash, the full development branch is available at:
https://github.com/cirosantilli/linux-kernel-module-cheat/tree/asm
This notably means a refactor of the userland build and testing, to support:
- improved assembly infrastructure unified across arm and x86
- native in-tree build and test helpers
- parallel building and testing, which implies thread_pool.py
- selection of what to build and test from the CLI
- path_properties.py to indicate how to build and run different examples
- in full system, move all userland stuff into /lkmc
- prefix everything that we defined across files with LKMC
- --gdb uber convenient helper
- remove import imp which was deprecated
Full commit messages from the branch follow:
1:
userland: add assembly support
Move arm assembly cheat here, and start some work on x86 cheat as well.
2:
document userland asm syscall interfaces
3:
userland assembly: structure readme
4:
x86 fail works
5:
asm: more links
6:
userland: add ported to all archs
7:
move all our stuff into /lkmc in guest
Motivation: userland is getting several new subdirectories, it would be
too insane to just dump all of that in the guest root filesystem.
To alleviate the cd pain, .profile puts user inside /lkmc by default.
8:
start the big userland migration
9:
migrate all
10:
bak
11:
build-userland-in-tree is now a Python command
./build calls it, we did this to allow --download-dependencies to work
perfectly.
12:
rename include to lkmc
13:
mtops.h is perfect now
14:
userland: make build perfect
15:
preparing test_user_mode, need to generalize stuff as usual
16:
asm: prefix every linux specific with linux/
17:
userland: maybe it really works
18:
userland: fix kernel version to work on older ubuntu
Expose --kernel-version to allow customization.
Update LTP info.
19:
userland: build really truly working now
userland test: start work, in a working state, but no features
20:
test-user-mode: make perfect like build-userland
Multithreading and target selection.
21:
userland: get a bit closer to perfection
22:
thread_pool: support passing thread IDs
Then use that to fix gem5 error log read race.
23:
userland: native testing
24:
userland: path properties getting nice!
25:
userland: move posix/environ from cpp-cheat
26:
gem5: --debug-flags without =, looks nicer whenever it can be done
27:
run: rename --wait-gdb in --gdb-wait, --gdb prefix might become a thing
28:
run: create --tmux-program gdb to open gem5 GDB
29:
run: create the uber convenient --gdb option
30:
userland: move getchar from cpp-cheat
31:
prebuilt: kernel boot aarch64 does not work on Ubuntu 16.04
32:
userland: x86_64 linux hello world make PIE
33:
userland: try to make userland executable selection saner
Only allow existing files to be built, stop extension expansion madness.
cli_function: get_cli print booleans properly, was printing without --no-
for negations.
34:
userland: only link to lkmc.o if needed
35:
path_properties: make data very compact with only tuples and dicts
Spend 2 hours of my life thinking about low value tree walks ;-)
36:
userland: move more userland/arch/ logic into property tree
37:
userland: make libs work
Working for build, but now test-user-mode-in-tree is not using --in-tree,
TODO fix later on.
38:
userland: make libs really work
39:
userland: document path_properties
40:
userland: classify linux
41:
waste your life
42:
common: fix absolute path runs
--gdb: allow running from arbitrary directory
43:
baremetal: arm allow using floating point instructions
44:
baremetal: stat preparing to make perfect like userland/
45:
run: fix image check logic accounting for userland
Was failing if I try to run userland (with abspath) when out/
directory is not present.
46:
cli-function: raise if the config file is given and does not exist
47:
common: define missing 'ld' variable, this broke m5 build
48:
rum: --qemu-which host now works for user mode as well as system
Don't fall back on host QEMU automatically, too much insanity.
49:
userland: refix silly mistakes
50:
userland: use path_properties flags for all builds, including lkmc. and userland/arch/main.c
Without this in particular, --gdb fails on assembly because main.c
was not being built with -ggdb3.
51:
userland: start refactor to show failing values on failure!
aarch64 basically done, but missing:
- other archs
- maybe convert main.c into C++ to use templates?
- full review of ASSERT_EQ calling convention issues not seen by tests
by chance
- documentation
52:
readme: releases are more stable...
53:
submodules: sort gitmodules
54:
test-baremetal: same interface as test-user-mode
In particular, runs tests in parallel, and allows selecting given tests
55:
baremetal: allow arbitrary exit status with the magic string
test-baremetal: fix missing setting x0 return value
Examples were just returning on ret without setting x0, which led to
failures... those were not noticed because of how broken the testing system
was ;-)
56:
baremetal: ah, actually nope, it didn't work :-(
Workaround for now. Works on asserts, but not on exit 1.
Some other day, maybe.
https://github.com/cirosantilli/linux-kernel-module-cheat/issues/59
57:
panic on panic: improve behaviour description
58:
baremetal: get exit status working with on_exit :-)
59:
baremetal: implement C assert
60:
test-baremetal: remove commented out exit status workaround
61:
test-user-mode: handle exit status for signals. Fix#61.
62:
aarch64: fix ASSERT_EQ_REG tests on gem5
Was doing an 8-byte aligned store, which gem5 dislikes.
But the ARMARM says bad things may happen there, notably a signal:
"D1.8.2 SP alignment checking" so gem5 is not really too wrong,
QEMU just happens to work by chance.
63:
userland assembly: build empty.S and fail.S to toplevel and run fail.S with path_properties exit_status
They were just duplicating stuff needlessly while we don't support non-native in-tree builds,
which leads to executable conflicts for C file anyways.
64:
gem5: use a single build tree for all build types
gem5 already has different object names for each build type it seems, so
let's just make sure that works and save some disk space.
65:
userland x86_64: ASSERT_EQ show actual and expected values
66:
assert_fail.c: add to readme index
67:
userland x86_64: implement ASSERT_MEMCMP
68:
userland x86_64: allow ASSERT_EQ to take just about anything
69:
gas data sizes
70:
gas_data_sizes.S: make PIE for all ISAs
71:
x86: paddq
72:
x86 paddq: test entire family
73:
Get rid of imp, started giving deprecation warning every time in Python 3.7 in Ubuntu 19.04.
Please python stop torturing me with refactors.
Make ./run -u blow up if executable not found, otherwise I go crazy.
Get ./test-gdb back to life after the ./run relative path refactor, forgot to test this.
74:
fix run-toolchain, qemu-monitor, trace-boot, trace2line, bisect-linux-boot-gem5. Fixes part of #63
I'm sad no one reported qemu-monitor break, that one is kind of important.
count.out arguments broke it as an init program, since the kernel adds trash
parameters to every init.
Is anyone using this repo, I wonder? Keep pushing, keep pushing.
One day it gets good enough, and the whole world will see.
75:
x86 assembly: addpd
76:
Fix import_path circular dependency by splitting it out.
Use import thread_pool instead from, from is evil.
Fix poweroff.out path for ./trace-boot.
77:
run: rename cryptic tmu to tmux-split, ./run is good now so I never use it anymore explicitly
78:
assembly SIMD add: make uniform for all ISAs, mark as entry point to learning SIMD
79:
start moving arm-assembly-cheat readme in here
80:
arm assembly: move some more in
81:
move more arm in
82:
userland: attempt to fix all assembly example links to README
83:
assembly: improve organization of simd add
84:
ld2 move in
85:
Make userland / assembly getting started more uniform / visible
Forward --gcc-which to ./run --tmux.
Use gdb-multiarch for --gcc-which host.
86:
userland: disable PIE explicitly on command line for all executables
87:
userland: make userland content a better landing page
88:
build: check git version from --version and degrade gracefully
89:
build: make --dry-run work again on all
90:
import_path: importlib explicit for Ubuntu 16.04
91:
make all submodules point to my forks
git servers are insane, submodule implementation is crap, what can you do
92:
build: log warning on git too old for --update
93:
build-linux: do olddefconfig even if no fragments
In particular, gem5 kernel 4.15 needs it
94:
userland content: improve a bit landing page for cpp-cheat
I was considering setting --static by default to match gem5, but then
that breaks shared libraries like openblas... so let's just use the
ugly workaround for now as it seems to work...
This is a squash commit, the unsquashed development went through many
unstable phases which would break bisects. The unsquashed branch is:
https://github.com/cirosantilli/linux-kernel-module-cheat/tree/v3.0-unsquash
The main improvement of this release was to greatly generalize the testing system.
The key addition was cli_function.py, which allows scripts such as ./run to
be transparently called either from Python or from the command line.
New tests scripts were created using this improved framework: test-baremetal
and test-user-mode.
We were lazy to port some of less important tests to the new setup, TODO's were
added, and we need comes they will be fixed. Getting started is however sacred
as usual and should work.
Other changes include:
- gem5: update to 7fa4c946386e7207ad5859e8ade0bbfc14000d91
- run: --tmux-args implies --tmux
- run: add --userland-args to make userland arguments across QEMU and gem5
Get rid of --userland-before as a consequence.
- bring initrd and initramfs back to life
- build-userland: create --static to make build a bit easier
- gem5: --gem5-worktree also set --gem5-build-id
- remove --gem5, use --emulator gem5 everywhere
Allow passing --emulator multiple times for transparent tests selection
just like --arch.
- test-userland: allow selecting just a few tests
- linux: update to v4.20
- buildroot: update to 2018.08
The main motivation for this was to fix the build for Ubuntu 18.10, which
has glibc 2.28, which broke the 2018.05 build at the m4-host package with:
#error "Please port gnulib fseeko.c to your platform!
- getvar --type input
- failed xen attempt, refactor timer, failed svc attempt, aarch64 use gicv3
- build-doc: exit 1 on error, add to release testing
- build: add --apt option to make things easier on other distros
- build-linux: --no-modules-install
This will allow for other types of root filesystems that don't rely on Buildroot
to be added and used in the future.
Propagate --verbose on all build scripts to see full GCC commands.
build-all: allow for neat subsets
also 9p share rootfs_overlay. TODO document.
Unsquashed version at v2-rc-unsquashed, but that cannot be merged as it
breaks bisects at several points. All bugs will not bisect to this
humongous change.
It all started with a conversion of the Bash scripts to Python, mainly
because I couldn't stand not being able to properly use --options for
run which has a million options.
Then since that required a full testing, I decided to do all the
refactorings that I had in mind at once, and so I did and it became
v2-rc.
This is the largest patch I have ever done! OMG a few weeks of extra time.
I'm never writing a Bash script for anything that starts getting big again.
Some of the features are:
* separate build-qemu and build-gem5 commands
* common: convert scripts to python. Add --option for everything
* rename build to build-buildroot now that we are splitting all the build
commands, Linux kernel to follow
* move all git submodules to submodules/ and all buildroot packages to
packages/
* refactor the out/ structure. Keep projects on toplevel, because guest
projects separate archs and host ones don't, making a toplevel arch wrong
* do-release: rename to just release
https://stackoverflow.com/questions/16174992/cant-get-argparse-to-read-quoted-string-with-dashes-in-it
* run: add --terminal and explain gem5 pdb
* just track the lvimrc
* store CLI kernel config fragment inside buildlroot to avoid conflicts
* gem5: document m5 initparam
* readme: make a bunch of things awesomer
* readme: fix broken refs
* parsec-benchmark: update to 75d55ac446a43c47efb1044844a108c6c330184c
Could not fetch otherwise.
* gem5: M5_OVERRIDE_PY_SOURCE
Otherwise, checking out branches is too insane, as it does not
update the worktrees, even though the gem5/gem5 module was updated.
gem5: expose build types, document debug builds.
simultaneous runs: store stdout and stderr on a file to allow running
all from a single terminal on the background cleanly.
Automatically pick up packages from under packages/ into BR2_EXTERNAL.
Move many subdirectory READMEs into the toplevel and link to toplevel from those subreadmes instead.
We have wanted to do this since forever, but the last straw was gem5 aarch64,
which runs too fast, and makes it very hard to write "root" withing 60 seconds!
Another possibility to solve that would have been to find an answer for:
https://unix.stackexchange.com/questions/340333/how-can-i-get-bin-login-to-not-timeout
which we didn't, but not typing root at all is even better.
Take this opportunity to document how to login into user0.
sshd: automate and document further
gdbserver: automatically startup internet on /gdbserver.sh
