Break out the tar parsing parts into a function for just that and create
a new one that knows how to load directly from the html directory in the
postgresql source tree, for more efficient snapshot loading.
This will require some further updates on the loading side of things
before it's fully valid, but for now track and show a link to the git
hash used to build developer docs *if* one is specified.
We only track it for devel (because releases have release numbers) and
we only show it in the cases where we would already show the loading
time.
The fix in 52455885 broke the data.sql preloading of example data.
Fix the insert for the main website (patch from Célestin Matte), and
also delete those parts of the file that are no longer relevant.
This table isn't used beyond a single row at this time, so there is a
lot to be done for simplifying things around it, but for a quick-fix
just add the field that actually should be there.
Spotted by Célestin Matte
Instead of printing every single page loaded, print the start of the
process and the statitics by default.
The existing --quiet parameter continues to work to make the process
completely quiet.
Add a new parameter --verbose that makes it run in the old way, printing
everything.
By calling the purge notifier script with --static, the paths wil be
taken as part of the pgweb-static repostory instead, and purged with a
prefix of files/
This hasn't worked for a few years (since 900946df) and nobody has
complained. Getting rid of it allows similifying a few things on the
search server side as well.
We already avoided the actual updates (by filtering the UPDATE
statement), but since we set the load date we'd trigger a change to
every page and kick it ouf of the caches even when not chagned, which is
wasteful. So instead only do that if something has changed. When it has,
we still reset that whole version of the docs since we want the load
date to be consistent across them.
Previously we used a combination of optimistic concurrency control
(by DELETEing with both the id and the date included in the WHERE
clause) and REPEATABLE READ transactions. This would create
serialization conflicts when completely unnecessary. Since in this case
it doesn't matter if we happen to push the same thing twice, switch
completely to optimistic concurrency control. That gets rid of having to
deal with serialization issues.
This should not be possible if things are done through the appropriate
interfaces, and those interfaces are bug-free (right..). But really bad
things can probably happen if they don't, so put a monitor in place to
check for it.
This also adds a view in the db that shows all registered email
addresses and their accounts, regardless of if it's primary or secondary
address. This is used by the nagios check but can of course be useful to
manual checks as well.
It really shouldn't exist, but we clearly have cases where the community
auth plugin sets the email to NULL, so we need to handle that case when
looking for what to update.
This adds the concept of an apiurl to each site that uses community
authentication, that the main website server can make calls to and send
updates. This URL will receive POSTs from the main website when a user
account that has been used on this site gets updated, and can then
optionally update it's local entries with it (the django plugin sample
is updated to handle this fully).
Updates are only sent for users that have a history of having logged
into the specific site -- this way we avoid braodcasting user
information to sites requiring specific constent that the user hasn't
given, and also decreases the amount of updates that have to be sent.
Updates are queued by the system in a table and using listen/notify a
daemon that's running picks up what needs to be updated and posts it to
the endpoints. If this daemon is not running, obviously nothing gets
sent.
Updates are tracked using triggers in the database which push
information into this queue.
It's useful to be able to directly get at the user record that was
returned instead of having to re-query it from the database. Since
nothing was previously returned, this is not backwards incompatible.
The yum project has switched to single-reporpm-for-multiple-versions as
well as putting them in a dedicated directory. This makes a lot of the
cruft in the code to deal with the distribution of reporpms unnecessary,
so change it.
We still need to keep track of which platforms exist for which versions,
so we do still need some spidering, but it's alot simpler.
Since all versions are also always availeble, the javascript that
populates the version list can now just use the list of supported
versions. This could be rendered directly in the HTML, but we might need
to restrict versions-per-platform at some point in the future, and
javascript is needed anyway for the rest of the page, so we keep doing
it the old way.
Issuing individual INSERTs for each line in the docs works decently when
local, but it slow when loading across The Tubes Of The Internet.
Switching to using COPY takes the load time from the buildfarm animal
from just over 2 minutes to about 6-7 seconds.
