* src/cuirass/metrics.scm (db-update-metrics): Update all metrics in a single
transaction.

Registering an evaluation can cause a large number of write queries using
individual transactions. This is really sub-optimal as commiting a transaction
will often result in a database sync to the disk.

* src/cuirass/database.scm (db-add-build): Do not create a new transaction for
each new build, instead ...
(db-register-builds): ..., create a single transaction per evaluation
registration.

* doc/cuirass.texi (Invoking cuirass): Update path for gcroots.

Signed-off-by: Mathieu Othacehe <othacehe@gnu.org>

* .dir-locals (scheme-mode): Update it.

* src/cuirass/metrics.scm: Add missing copyright.
* src/cuirass/watchdog.scm: Ditto.

Make sure that db-get-builds only performs one query and add build products to
the result. Also add indexes so that all "order" filters are covered.

Limit the maximum of build objects returned in "/api/latestbuilds" and
"/api/queue" to 1000.

Fixes: <https://issues.guix.gnu.org/43826>.

* src/sql/upgrade-15.sql: New file.
* Makefile (dist_sql_DATA): Add it.
* src/cuirass/database.scm (db-get-builds-by-search): Add "buildproducts" to
the returned build objects.
(db-get-builds): Remove unused filters. Join Outputs and BuildProducts table
in the query. Add "buildproducts" to the returned build objects.
* src/cuirass/http.scm (build->hydra-build): Use "buildproducts" from
"db-get-builds" procedure.
(url-handler): Forbid limit parameters greater than 1000 and use
"buildproducts" from "db-get-builds" procedure.
* src/schema.sql (Builds_stoptime, Builds_stoptime_id, Builds_status_ts_id):
New indexes.

* src/sql/upgrade-14.sql: New file.
* src/schema.sql (dist_sql_DATA): Add it.
* src/schema.sql (Builds_timestamp_stoptime): New index.

* src/cuirass/database.scm: Remove unused modules.

Searching for something like "%a%" can take up to ten minutes. Remove the
usage of special characters "^" and "$" in queries. Instead, always add a "%"
character at the end of the query. Also add an index on the nix_name field.

Fixes: <https://issues.guix.gnu.org/43791>.

* src/sql/upgrade-13.sql: New file.
* Makefile.am (dist_sql_DATA): Add it.
* src/schema.sql (Builds_nix_name): New index.
* src/cuirass/database.scm (query->bind-arguments): Remove support for "^" and
"$" special characters. Instead make sure that the query does not contain any
"%" character and prefix the query by "%".
(db-get-builds-by-search): Remove an useless query nesting level.
* src/cuirass/templates.scm (search-form): Adapt the search help message
accordingly.

This is a follow-up of 761443bc.

* src/cuirass/base.scm (fetch-inputs): Use map to replace the previous par-map
call.
(compile-checkouts): Also replace par-map call by map.

Fixes: <https://issues.guix.gnu.org/43757>.

* src/cuirass/base.scm (fetch-inputs): Call par-map from a separate thread as
this call is not suspendable.

The number of database worker is limited, and can introduce some
starvation. Print a message each time a two seconds timeout expires while
sending an SQL query to a worker.

* src/cuirass/utils.scm (put-message-with-timeout): New procedure.
(call-with-worker-thread): Rename timeout and timeout-proc arguments to
receive-timeout and receive-timeout-proc arguments. Add two new arguments:
send-timeout and send-timeout-proc. Use put-message-with-timeout to honor
them.
* src/cuirass/database.scm (with-db-worker-thread): Adapt accordingly and pass
send-timeout and send-timeout-proc arguments.

* src/cuirass/base.scm (build-packages): Improve the logging message.

* src/cuirass/base.scm (build-packages): Print the id of the registered
evaluation.

* src/cuirass/database.scm (with-db-worker-thread): Print the caller name.

* src/cuirass/logging.scm (log-query): Add date.

* src/cuirass/database.scm (db-register-builds): Remove store argument and
assume that 'log and 'outputs properties are provided by the evaluation.
* src/cuirass/base.scm (build-packages): Adapt accordingly.

Evaluation registration involves running a large number of SQL queries. This
can cause some database worker starvation as well as some contention. To avoid
this issue, spawn database workers dedicated to evaluation registration.

* src/cuirass/database.scm (%db-registration-channel): New variable.
(with-db-registration-worker-thread, with-registration-workers): New macros.
(with-db-worker-thread-no-timeout): Remove it.
(db-register-builds): Run registration in dedicated database workers using
"with-db-registration-worker-thread" macro.
* bin/cuirass.in (main): Spawn database registration workers by calling
"with-registration-workers" macro.

Make sure that all registration queries are done from within a single database
worker. Otherwise, when builds from multiple evaluations are registered at
the same time, some contention occurs communicating with workers.

* src/cuirass/base.scm (new-outputs?, build-packages): Move build registration
to ...
* src/cuirass/database.scm (db-register-builds): ... this new procedure.
(with-db-worker-thread-no-timeout): New procedure. Use it in
"db-register-builds" to avoid timeout messages.

* src/cuirass/base.scm (evaluate): Expect 'nix-name and 'system properties to
be provided by the evaluation.
(build-packages): Use the eval-id provided as argument.

Avoid full Builds table scans.

* src/cuirass/metrics.scm (db-average-build-start-time-per-eval,
db-average-build-complete-time-per-eval, db-evaluation-completion-speed):
Rewrite query to avoid full Builds table scans.

* src/cuirass/watchdog.scm: New file.
* Makefile.am (dist_pkgmodule_DATA): Add it.
* src/cuirass/utils.scm (with-timeout, get-message-with-timeout): Export them.
* bin/cuirass.in (main): Start the watchdog.

Avoid full scans of Builds table that can be very time consuming by rewriting
some queries and using new indexes.

* src/sql/upgrade-12.sql: New file.
* Makefile.am (dist_sql_DATA): Add it.
* src/schema.sql (Builds_evaluation_index, Evaluations_status_index,
Evaluations_specification_index): New indexes.
* src/cuirass/database.scm (db-get-evaluations-build-summary,
db-get-evaluation-summary): Rewrite queries to avoid full Builds table scan
and use the new indexes.

Once the evaluation is over, the new builds are registered. This registration
tries to insert a new build for each derivation returned by the evaluation
phase. If the new build does not add a new output, the insertion query is then
rollbacked. This means that there are at least as many insertion queries as
new derivations.

SQlite allows at most one writer at a time, and even though we are using WAL
mode, performing a lot of insertions will reduce the reading perforances.
When multiple evaluations are performed in parallel, the large number of
concurrent insertion queries also causes contention.

To avoid those issues, check first in the "Outputs" table which derivations
are already registered. This means that most of the insertion queries will be
replaced by reading queries, that are much less expensive and more suitable
for Cuirass concurrent implementation.

* src/cuirass/base.scm (new-outputs?): New procedure.
(build-packages): Use it to insert only builds registering new outputs.

On fresh Guix System installations, Cuirass will not be allowd to create its
own profile directory. Instead this should be take care of by guix-daemon when
Cuirass will initiate its first connection to it.

* bin/cuirass.in (main): Do not raise an exception on gcroot directory
creation failure.

* bin/cuirass.in (show-help): Document "--log-queries" option.
(%options): Add it.
(main): Enable query logging if the above option is set.
* src/cuirass/database.scm (db-log-queries): New procedure.
* src/cuirass/logging.scm (query-logging-port): New parameter.
(log-query): New procedure.

If a derivation is GC'd just before calling "spawn-builds",
"build-derivations: will throw an exception that will be catched. Then,
"update-build-statuses!"  will call "derivation-path->output-paths" that will
throw another exception because the derivation does not exit. This exception
is not handled, causing Cuirass to crash.

2020-09-18T10:41:18 batch of builds (partially) failed: build of `/gnu/store/zrmxzjf025nc89a7vdy5i94zavprc7fs-emacs-guix-0.5.2-2.58a840d.drv' failed (status: 1)
Backtrace:
In ice-9/boot-9.scm:
  1736:10 11 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _)
In unknown file:
          10 (apply-smob/0 #<thunk 7ffff5cbd4a0>)
In ice-9/boot-9.scm:
    718:2  9 (call-with-prompt _ _ #<procedure default-prompt-handler (k proc)>)
In ice-9/eval.scm:
    619:8  8 (_ #(#(#<directory (guile-user) 7ffff58f8f00>)))
In ice-9/boot-9.scm:
   2806:4  7 (save-module-excursion _)
  4351:12  6 (_)
In cuirass/base.scm:
   562:10  5 (spawn-builds #<store-connection 256.99 7fffec03f910> _ #:max-batch-size _)
In srfi/srfi-1.scm:
    634:9  4 (for-each #<procedure update! (drv)> ("/gnu/store/zrmxzjf025nc89a7vdy5i94zavprc7fs-emacs-guix-0.5.2-2.58a840d.drv"))
In cuirass/base.scm:
    474:4  3 (update! "/gnu/store/zrmxzjf025nc89a7vdy5i94zavprc7fs-emacs-guix-0.5.2-2.58a840d.drv")
In guix/derivations.scm:
   552:17  2 (derivation-path->output-paths "/gnu/store/zrmxzjf025nc89a7vdy5i94zavprc7fs-emacs-guix-0.5.2-2.58a840d.drv")
In ice-9/ports.scm:
   440:11  1 (call-with-input-file "/gnu/store/zrmxzjf025nc89a7vdy5i94zavprc7fs-emacs-guix-0.5.2-2.58a840d.drv" #<procedure read-derivation (drv-port #:optional read-derivation-from-file)> #:binary _ …)
In unknown file:
           0 (open-file "/gnu/store/zrmxzjf025nc89a7vdy5i94zavprc7fs-emacs-guix-0.5.2-2.58a840d.drv" "r" #:encoding #f #:guess-encoding #f)

* src/cuirass/base.scm (update-build-statuses!): Catch
"derivation-path->output-paths" exceptions and set the build status to
"failed".

* src/cuirass/http.scm (metrics-page): Order 'average-eval-build-start-time
and 'evaluation-completion-speed metrics per ascending evaluation id.

* src/cuirass/templates.scm (global-metrics-content): Rename "Builds speed"
section to "Builds completion".

* tests/metrics.scm ("builds-per-day", "new-derivations-per-day"): Fix tests
results that were depending of the local time.
("evaluation-completion-speed"): Adapt to new unit.

* src/cuirass/templates.scm (global-metrics-content): Add legends.

* src/cuirass/templates.scm (make-line-chart): Do not display chart title.

* src/cuirass/http.scm (url-handler): Move metric page creation to ...
(metrics-page): ... this new procedure. Pass average evaluation build start
time and evaluation completion speed to "global-metrics-content".
* src/cuirass/templates.scm (make-line-chart): Add "time-x-axes?",
"xaxes-labels", "x-label" and "y-label" arguents.
(global-metrics-content): Add "avg-eval-build-start-time" and
"eval-completion-speed" arguments. Create and display two new charts from
those metrics.

* src/cuirass/metrics.scm (db-evaluation-completion-speed): Change the
completion speed unit to builds/hour.
(%metrics): Adapt comment.

* src/cuirass/metrics.scm (db-average-build-start-time-per-eval,
db-average-build-complete-time-per-eval, db-evaluation-completion-speed,
db-latest-evaluations): New procedures.
(%metrics): Add 'average-eval-build-start-time,
'average-eval-build-complete-time, 'evaluation-completion-speed.
(db-update-metrics): Update evaluation related metrics for the evaluations
added the past three days.
* tests/metrics.scm ("average-eval-build-start-time",
"average-eval-build-complete-time", "evaluation-completion-speed"): Add new
tests.

* bin/cuirass.in (main): Log metrics update duration.

* tests/metrics.scm (nearest-exact-integer): Remove as unused.
("db-update-metrics"): Test that 'pending-builds metric is updated on
"db-update-metrics" call.

* tests/metrics.scm: New file.
* Makefile.am (TESTS): Add it.

* src/cuirass/metrics.scm (db-percentage-failed-eval-per-spec): New procedure.
(%metrics): Add 'percentage-failure-10-last-eval-per-spec,
'percentage-failure-100-last-eval-per-spec and
'percentage-failed-eval-per-spec metrics.
(db-update-metrics): Add them.
* src/cuirass/templates.scm (global-metrics-content): Add
"percentage-failed-eval" argument. Use it to display the "percentage of failed
evaluations" table.
* src/cuirass/templates.scm (url-handler): Adapt accordingly.

* src/cuirass/metrics.scm (db-average-eval-duration-per-spec): Order results
by descending rowid and adjust docstring.