Avoid full scans of Builds table that can be very time consuming by rewriting
some queries and using new indexes.

* src/sql/upgrade-12.sql: New file.
* Makefile.am (dist_sql_DATA): Add it.
* src/schema.sql (Builds_evaluation_index, Evaluations_status_index,
Evaluations_specification_index): New indexes.
* src/cuirass/database.scm (db-get-evaluations-build-summary,
db-get-evaluation-summary): Rewrite queries to avoid full Builds table scan
and use the new indexes.

Once the evaluation is over, the new builds are registered. This registration
tries to insert a new build for each derivation returned by the evaluation
phase. If the new build does not add a new output, the insertion query is then
rollbacked. This means that there are at least as many insertion queries as
new derivations.

SQlite allows at most one writer at a time, and even though we are using WAL
mode, performing a lot of insertions will reduce the reading perforances.
When multiple evaluations are performed in parallel, the large number of
concurrent insertion queries also causes contention.

To avoid those issues, check first in the "Outputs" table which derivations
are already registered. This means that most of the insertion queries will be
replaced by reading queries, that are much less expensive and more suitable
for Cuirass concurrent implementation.

* src/cuirass/base.scm (new-outputs?): New procedure.
(build-packages): Use it to insert only builds registering new outputs.

On fresh Guix System installations, Cuirass will not be allowd to create its
own profile directory. Instead this should be take care of by guix-daemon when
Cuirass will initiate its first connection to it.

* bin/cuirass.in (main): Do not raise an exception on gcroot directory
creation failure.

* bin/cuirass.in (show-help): Document "--log-queries" option.
(%options): Add it.
(main): Enable query logging if the above option is set.
* src/cuirass/database.scm (db-log-queries): New procedure.
* src/cuirass/logging.scm (query-logging-port): New parameter.
(log-query): New procedure.

If a derivation is GC'd just before calling "spawn-builds",
"build-derivations: will throw an exception that will be catched. Then,
"update-build-statuses!"  will call "derivation-path->output-paths" that will
throw another exception because the derivation does not exit. This exception
is not handled, causing Cuirass to crash.

2020-09-18T10:41:18 batch of builds (partially) failed: build of `/gnu/store/zrmxzjf025nc89a7vdy5i94zavprc7fs-emacs-guix-0.5.2-2.58a840d.drv' failed (status: 1)
Backtrace:
In ice-9/boot-9.scm:
  1736:10 11 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _)
In unknown file:
          10 (apply-smob/0 #<thunk 7ffff5cbd4a0>)
In ice-9/boot-9.scm:
    718:2  9 (call-with-prompt _ _ #<procedure default-prompt-handler (k proc)>)
In ice-9/eval.scm:
    619:8  8 (_ #(#(#<directory (guile-user) 7ffff58f8f00>)))
In ice-9/boot-9.scm:
   2806:4  7 (save-module-excursion _)
  4351:12  6 (_)
In cuirass/base.scm:
   562:10  5 (spawn-builds #<store-connection 256.99 7fffec03f910> _ #:max-batch-size _)
In srfi/srfi-1.scm:
    634:9  4 (for-each #<procedure update! (drv)> ("/gnu/store/zrmxzjf025nc89a7vdy5i94zavprc7fs-emacs-guix-0.5.2-2.58a840d.drv"))
In cuirass/base.scm:
    474:4  3 (update! "/gnu/store/zrmxzjf025nc89a7vdy5i94zavprc7fs-emacs-guix-0.5.2-2.58a840d.drv")
In guix/derivations.scm:
   552:17  2 (derivation-path->output-paths "/gnu/store/zrmxzjf025nc89a7vdy5i94zavprc7fs-emacs-guix-0.5.2-2.58a840d.drv")
In ice-9/ports.scm:
   440:11  1 (call-with-input-file "/gnu/store/zrmxzjf025nc89a7vdy5i94zavprc7fs-emacs-guix-0.5.2-2.58a840d.drv" #<procedure read-derivation (drv-port #:optional read-derivation-from-file)> #:binary _ …)
In unknown file:
           0 (open-file "/gnu/store/zrmxzjf025nc89a7vdy5i94zavprc7fs-emacs-guix-0.5.2-2.58a840d.drv" "r" #:encoding #f #:guess-encoding #f)

* src/cuirass/base.scm (update-build-statuses!): Catch
"derivation-path->output-paths" exceptions and set the build status to
"failed".

* src/cuirass/http.scm (metrics-page): Order 'average-eval-build-start-time
and 'evaluation-completion-speed metrics per ascending evaluation id.

* src/cuirass/templates.scm (global-metrics-content): Rename "Builds speed"
section to "Builds completion".

* tests/metrics.scm ("builds-per-day", "new-derivations-per-day"): Fix tests
results that were depending of the local time.
("evaluation-completion-speed"): Adapt to new unit.

* src/cuirass/templates.scm (global-metrics-content): Add legends.

* src/cuirass/templates.scm (make-line-chart): Do not display chart title.

* src/cuirass/http.scm (url-handler): Move metric page creation to ...
(metrics-page): ... this new procedure. Pass average evaluation build start
time and evaluation completion speed to "global-metrics-content".
* src/cuirass/templates.scm (make-line-chart): Add "time-x-axes?",
"xaxes-labels", "x-label" and "y-label" arguents.
(global-metrics-content): Add "avg-eval-build-start-time" and
"eval-completion-speed" arguments. Create and display two new charts from
those metrics.

* src/cuirass/metrics.scm (db-evaluation-completion-speed): Change the
completion speed unit to builds/hour.
(%metrics): Adapt comment.

* src/cuirass/metrics.scm (db-average-build-start-time-per-eval,
db-average-build-complete-time-per-eval, db-evaluation-completion-speed,
db-latest-evaluations): New procedures.
(%metrics): Add 'average-eval-build-start-time,
'average-eval-build-complete-time, 'evaluation-completion-speed.
(db-update-metrics): Update evaluation related metrics for the evaluations
added the past three days.
* tests/metrics.scm ("average-eval-build-start-time",
"average-eval-build-complete-time", "evaluation-completion-speed"): Add new
tests.

* bin/cuirass.in (main): Log metrics update duration.

* tests/metrics.scm (nearest-exact-integer): Remove as unused.
("db-update-metrics"): Test that 'pending-builds metric is updated on
"db-update-metrics" call.

* tests/metrics.scm: New file.
* Makefile.am (TESTS): Add it.

* src/cuirass/metrics.scm (db-percentage-failed-eval-per-spec): New procedure.
(%metrics): Add 'percentage-failure-10-last-eval-per-spec,
'percentage-failure-100-last-eval-per-spec and
'percentage-failed-eval-per-spec metrics.
(db-update-metrics): Add them.
* src/cuirass/templates.scm (global-metrics-content): Add
"percentage-failed-eval" argument. Use it to display the "percentage of failed
evaluations" table.
* src/cuirass/templates.scm (url-handler): Adapt accordingly.

* src/cuirass/metrics.scm (db-average-eval-duration-per-spec): Order results
by descending rowid and adjust docstring.

* src/cuirass/metrics.scm: Add a warning about metric insertion in %metrics
list.

* src/cuirass/metrics.scm (db-new-derivations-previous-day): New procedure.
(%metrics): Add 'new-derivations-per-day.
(db-update-metrics): Add it.
* src/cuirass/templates.scm (make-line-chart): Add support for multiple
datasets. Also add "interpolation?" and "legend?" options.
(global-metrics-content): Add "new-derivations-per-day" argument. Adapt
"Builds per day" chart so that it uses two datasets from
'new-derivations-per-day and 'builds-per-day metrics.
* src/cuirass/templates.scm (url-handler): Adapt accordingly.

* src/cuirass/metrics.scm (db-builds-previous-day): Only take into account the
builds that are created and processed during the previous day.

* src/cuirass/metrics.scm (db-pending-builds, db-current-day-timestamp): New
procedures.
(%metrics): Add 'pending-builds.
(db-update-metrics): Add it.
* src/cuirass/templates.scm (global-metrics-content): Add "pending-builds"
argument. Display pending builds in a line chart.
* src/cuirass/http.scm (url-handler): Adapt "global-metrics-content" call
accordingly.

* src/cuirass/templates.scm (make-line-chart): Do not override onload callback
as that would prevent us from adding multiple charts.

* src/cuirass/metrics.scm (%metrics): Rename 'builds-previous-day to
'builds-per-day.
(db-update-metrics): Adapt accordingly.
* src/cuirass/http.scm (url-handler): Ditto.

* src/cuirass/metrics.scm (db-get-metrics-with-id): Order by descending rowid
by default.
* src/cuirass/http.scm (url-handler): Use the default order for
"db-get-metrics-with-id".

* Makefile.am: Add "upgrade-11.sql", "chart.js" and "metrics.js".
* bin/cuirass.in (main): Add 'metrics fiber.
* src/cuirass/http.scm (%file-white-list): Add "js/chart.js".
(url-handler): Add "/metrics" route.
* src/cuirass/metrics.scm: New file.
* src/cuirass/templates.scm (html-page): Add a "Global metrics" sub-menu to
"Status" dropdown menu.
(make-line-chart, global-metrics-content): New procedures.
* src/schema.sql (Metrics): New table.
* src/sql/upgrade-11.sql: New file.
* src/static/js/chart.js: New file.

* src/cuirass/database.scm (last-insert-rowid, expect-one-row, %sqlite-exec):
Export them.

* Makefile.am (dist_sql_DATA): Add upgrade-9.sql and upgrade-10.sql.

This is a follow-up of 154232bc.

* tests/http.scm (evaluations-query-result): Add 'timestamp', 'checkouttime'
and 'evaltime' fields.
(fill-db): Pass a matching timestamp to "db-add-evaluation" calls.

The Evaluation table currently has an 'in_progress' field. Distinction between
succeeded and failed evaluations are based on the presence of Builds records
for the evaluation. It it also not possible to distinguish aborted evaluations
from failed evaluations.

Rename 'in_progress' field to 'status'. The 'status' field can be equal to
'started', 'succeeded', 'failed' or 'aborted'.

* src/cuirass/database.scm (evaluation-status): New exported enumeration.
(db-set-evaluations-done, db-set-evaluation-done): Remove them.
(db-abort-pending-evaluations, db-set-evaluation-status): New exported procedures.
(db-add-evaluation, db-get-builds, db-get-evaluations,
db-get-evaluations-build-summary, db-get-evaluation-summary): Adapt to use
'status' field instead of 'in_progress' field.
* src/cuirass/templates.scm (evaluation-badges): Ditto.
* src/schema.sql (Evaluations): Rename 'in_progress' field to 'status'.
* src/sql/upgrade-10.sql: New file.
* bin/cuirass.in (main): Use "db-abort-pending-evaluations" instead of
"db-set-evaluations-done".
* src/cuirass/base.scm (evaluate): Use "db-set-evaluation-status" instead of
"db-set-evaluations-done".
(build-packages): Use "db-set-evaluation-status" instead of
"db-set-evaluation-done".
* tests/database.scm (sqlite-exec): Adapt accordingly.
* tests/http.scm (evaluations-query-result): Ditto.

Commit time can be far away from the actual push time. Thus, use the checkout
time instead of the commit time to fill Checkouts 'timestamp' field.

* src/cuirass/base.scm (fetch-input): Return the checkout time instead of the
commit time.

This is a follow-up of d9879583.

* src/cuirass/templates.scm (evaluation-build-table): Do not print evaluation
date if the "timestamp" field equals zero.

src/cuirass/database.scm (db-get-evaluations): Add support for "timestamp",
"checkouttime" and "evaltime" fields,
(db-get-evaluation-summary): ditto.
src/cuirass/templates.scm (nearest-exact-integer, seconds->string): New
procedures,
(evaluation-build-table): print evaluation date and duration.

src/cuirass/base.scm (fetch-input): Add the commit timestamp to the returned
association list,
(process-specs): Pass a timestamp taken at procedure start and another one
taken after inputs are fetched to "db-add-evaluation" procedure. Once the
evaluation is over, call "db-set-evaluation-time" to save the evaluation
completion time.
src/cuirass/database.scm (db-set-evaluation-time): New procedure,
(db-add-checkout): Handle the "timestamp" field,
(db-add-evaluation): add "checkouttime" and "evaltime" arguments. Modify the
associated SQL query accordingly.
(db-get-builds): Use "Builds.timestamp" instead of "timestamp" as this field
is also part of the Evaluations table.
src/schema.sql (Checkouts): Add "timestamp" field,
(Evaluations): add "timestamp", "checkouttime" and "evaltime" fields.
src/sql/upgrade-9.sql: New file.
tests/database.scm (sqlite-exec): Adapt Evaluations table insertions to include
"timestamp", "checkouttime" and "evaltime" required fields.

Fixes <https://issues.guix.gnu.org/43163>.

* src/cuirass/http.scm (specification->json-object): New procedure,
(url-handler): use it for "/specifications" route to convert specification
objects into a representation suitable for json->scm.
* tests/http.scm ("/specifications"): Test the above route.

* configure.ac: Add Guile-zlib dependency.

* README (Requirements): Add Guile-zlib.
* build-aux/guix.scm: Ditto.
* src/cuirass/base.scm: Use Guile-zlib instead of (guix zlib).

Use the pre-unwind-handler of a catch clause instead of nesting
with-throw-handler inside a catch clause.

* src/web/server/fiberized.scm (with-ignored-disconnects): Remove it,
(client-loop): replace "with-ignored-disconnects" with a catch clause.

EPIPE and ECONNRESET errors are already ignored when thrown by "sendfile" and
"put-bytevector" procedures. It turns out "peek-char" can also cause such
errors. So ignore such errors within the whole procedure.

* src/web/server/fiberized.scm (client-loop): Wrap the whole procedure inside
"with-ignored-disconnects" instead of wrapping individually "sendfile" and
"put-bytevector" calls.

This is a follow-up of 2094d680.

* src/cuirass/http.scm (request-parameters): Fix typo.