* src/cuirass/remote.scm: New file.
* src/cuirass/remote-server.scm: New file.
* src/cuirass/remote-worker.scm: New file.
* bin/remote-server.in: New file.
* bin/remote-worker.in: New file.
* Makefile.am (bin_SCRIPTS): Add new binaries,
(dist_pkgmodule_DATA): add new files,
(EXTRA_DIST): add new binaries,
(bin/remote-server, bin/remote-worker): new targets.
* .gitignore: Add new binaries.
* bin/cuirass.in (%options): Add "--build-remote" option,
(show-help): document it,
(main): honor it.
* src/cuirass/base.scm (with-build-offload-thread): New macro,
(%build-remote?, %build-offload-channel): new parameters,
(make-build-offload-thread): new procedure,
(build-derivations/offload): new procedure,
(restart-builds): use it to offload builds when %build-remote? is set,
(build-packages): ditto.

src/cuirass/base.scm (build-packages): Remove debug message that can be
inaccurate if queue-size is superior to 1.

* src/sql/upgrade-17.sql: New file.
* Makefile.am (dist_sql_DATA): Add it.
* src/schema.sql (Builds): Add "machine" field.
* src/cuirass/database.scm (db-update-build-machine!): New procedure.
* src/cuirass/base.scm (handle-build-event): Use it to record the machine
performing the build.

* bin/cuirass.in (%options): Add "queue-size" argument.
(show-help): Document it.
(main): Honor it.
* src/cuirass/database.scm (%db-writer-queue-size): New parameter,
(with-queue-writer-worker): use it.

This is a follow-up of ad44a9bb.

* tests/http.scm (build-query-result): Add "evaluation" field.

* src/cuirass/database.scm (db-get-evaluation): New procedure.
* src/cuirass/http.scm (url-handler): Add "/api/evaluation" route.

* src/cuirass/http.scm (build->hydra-build): Add "evaluation" field.

Rather than calling sqlite-reset before reading the rows, call sqlite-reset
after. I think this is important to stop SQLite getting stuck because the
statement is still live even though it's not being read from after these
procedures return.

Assuming I've got the ordering right, I think using sqlite-fold-right directly
is simpler code as well.

* src/cuirass/database.scm (db-get-builds-by-search, format-build-products,
db-get-events): Rewrite fetching result rows.

sqlite-prepare will reset cached statements before returning them, so these
sqlite-reset calls are redundant.

* src/cuirass/database.scm (db-get-builds-query-min, db-get-builds-query-max):
Remove sqlite-reset calls.

All these statements are cached, but that's pointless if they're finalized
after use, because they'll need recreating on next use.

* src/cuirass/database.scm (%sqlite-exec, db-get-builds-query-min,
db-get-builds-query-max): Replace sqlite-finalize with sqlite-reset.

* src/cuirass/database.scm (db-get-builds): Fix pagination.

* src/cuirass/database.scm (db-get-builds-min, db-get-builds-max): Rewrite SQL
queries without nesting.

When the periodic argument is passed to a job, it will be registered only if
the time difference between the current time and the registration time of the
last build is greater than the specified period. Make sure that the last job
that is searched is part of the same specification.

* src/sql/upgrade-16.sql: New file.
* Makefile.am (dist_sql_DATA): Add it.
* src/schema.sql (Builds_job_name_timestamp): New index.
* src/cuirass/database.scm (db-get-time-since-previous-build): Add
"specification" argument,
(db-register-builds): pass it.

This is a work-around: https://issues.guix.gnu.org/44742.

* src/cuirass/base.scm (fetch-inputs): Set "current-error-port" parameter to a
void port to prevent the fetch progression bar to appear.

* src/cuirass/database.scm (db-get-time-since-previous-build): New procedure,
(db-register-builds): if the period argument is set, only register builds
which last registration is older than the specified period.

This reverts commit 38790945 that causes:

2020-11-02T11:05:08 fatal: uncaught exception 'wrong-type-arg' in 'build' fiber!
2020-11-02T11:05:08 exception arguments: ("struct-vtable" "Wrong type argument in position 1 (expecting struct): ~S" (#f) (#f))
In ice-9/boot-9.scm:
  1731:15 12 (with-exception-handler #<procedure 7fb1a93f9930 at ic…> …)
  1736:10 11 (with-exception-handler _ _ #:unwind? _ # _)
    718:2 10 (call-with-prompt ("break") #<procedure 7fb1ab76f440 a…> …)
    718:2  9 (call-with-prompt ("continue") #<procedure 7fb1ab77084…> …)
In ice-9/eval.scm:
    619:8  8 (_ #(#(#<directory (guile-user) 7fb1ac680f00> #<var…> …)))
In srfi/srfi-1.scm:
    634:9  7 (for-each #<procedure 7fb1a9525900 at cuirass/base.scm…> …)
In ice-9/boot-9.scm:
  1731:15  6 (with-exception-handler #<procedure 7fb1a95a94e0 at ic…> …)
  1669:16  5 (raise-exception _ #:continuable? _)
  1764:13  4 (_ #<&compound-exception components: (#<&assertion-fail…>)
In cuirass/utils.scm:
    319:8  3 (_ _ . _)
In ice-9/boot-9.scm:
  1731:15  2 (with-exception-handler #<procedure 7fb1ab2e3720 at ic…> …)
In cuirass/utils.scm:
   320:22  1 (_)
In unknown file:
           0 (make-stack #t)
ERROR: In procedure make-stack:
In procedure struct-vtable: Wrong type argument in position 1 (expecting struct): #f

Set the store socket as non-blocking so that fibers communicating with the
store don't get blocked as described here: https://issues.guix.gnu.org/43565.

* src/cuirass/base.scm (with-store): Set the store socket as non-blocking.
(build-derivations&): Unset current-read-waiter and current-write-waiter.

* src/cuirass/database.scm (db-register-builds): Use
"with-db-writer-worker-thread/force" macro.

* .dir-locals.el: Add "with-queue-writer-worker".
* bin/cuirass.in: Modify "with-queue-writer-worker" scope to include the
web-server operations.
* src/cuirass/database.scm (with-db-writer-worker-thread): Export it.
(with-db-writer-worker-thread/force): New macro.
(db-add-input, db-add-checkout, db-add-specification, db-remove-specification,
db-add-evaluation, db-abort-pending-evaluations, db-set-evaluation-status,
db-set-evaluation-time, db-add-output, db-add-build-product, db-add-event,
db-delete-events-with-ids-<=-to): Use "with-db-writer-worker-thread" or
"with-db-writer-worker-thread/force" instead of "with-db-worker-thread".
* src/cuirass/metrics.scm (db-update-metrics): Ditto.
* tests/database.scm ("db-init"): Set "%db-writer-channel".
* tests/http.scm ("db-init"): Ditto.
* tests/metrics.scm ("db-init"): Ditto.

* src/cuirass/base.scm (cancel-old-builds): Remove it.

This is a follow-up of b67f38a7.

* src/cuirass/database.scm (catch-sqlite-error): New macro.
(SQLITE_CONSTRAINT_PRIMARYKEY, SQLITE_CONSTRAINT_UNIQUE, %db-writer-channel):
New variables.
* tests/database.scm (with-temporary-database): Set "%db-writer-channel".
(db-add-build-with-fixed-output): Catch sqlite error.
(db-get-pending-derivations): Do not add builds with duplicated outputs.

* src/cuirass/database.scm (set-db-options): Optimize database parameters.

SQLite only allows one concurrent write query operation. Having multiple
database workers calling "db-update-build-status!", will thus increase worker
starvation. Every write operation will also be done is a single transaction.

For those reasons, create a database worker dedicated to write queries. Have
this worker queue work and issue all the queued work queries in a single
transaction.

* .dir-locals.el: Add with-db-writer-worker-thread.
* src/cuirass/database.scm (with-queue-writer-worker): Rename
"with-registration-workers" macro.
(%db-writer-channel): Rename "%db-registration-channel" variable.
(with-queue-writer-worker): Rename "with-registration-workers".
(db-register-builds): Use "with-db-writer-worker-thread" instead of
"with-db-registration-worker-thread".
(db-update-build-status!): Ditto
* src/cuirass/utils.scm (make-worker-thread-channel): Add "queue-size" and
"queue-proc" arguments.
(call-with-worker-thread): Add "options" argument.
* bin/cuirass.in (main): Use "with-queue-writer-worker" instead of
"with-registration-workers". Modify the macro scope to include all the
possible write operations.

* src/cuirass/metrics.scm (db-update-metrics): Update all metrics in a single
transaction.

Registering an evaluation can cause a large number of write queries using
individual transactions. This is really sub-optimal as commiting a transaction
will often result in a database sync to the disk.

* src/cuirass/database.scm (db-add-build): Do not create a new transaction for
each new build, instead ...
(db-register-builds): ..., create a single transaction per evaluation
registration.

* doc/cuirass.texi (Invoking cuirass): Update path for gcroots.

Signed-off-by: Mathieu Othacehe <othacehe@gnu.org>

* .dir-locals (scheme-mode): Update it.

* src/cuirass/metrics.scm: Add missing copyright.
* src/cuirass/watchdog.scm: Ditto.

Make sure that db-get-builds only performs one query and add build products to
the result. Also add indexes so that all "order" filters are covered.

Limit the maximum of build objects returned in "/api/latestbuilds" and
"/api/queue" to 1000.

Fixes: <https://issues.guix.gnu.org/43826>.

* src/sql/upgrade-15.sql: New file.
* Makefile (dist_sql_DATA): Add it.
* src/cuirass/database.scm (db-get-builds-by-search): Add "buildproducts" to
the returned build objects.
(db-get-builds): Remove unused filters. Join Outputs and BuildProducts table
in the query. Add "buildproducts" to the returned build objects.
* src/cuirass/http.scm (build->hydra-build): Use "buildproducts" from
"db-get-builds" procedure.
(url-handler): Forbid limit parameters greater than 1000 and use
"buildproducts" from "db-get-builds" procedure.
* src/schema.sql (Builds_stoptime, Builds_stoptime_id, Builds_status_ts_id):
New indexes.

* src/sql/upgrade-14.sql: New file.
* src/schema.sql (dist_sql_DATA): Add it.
* src/schema.sql (Builds_timestamp_stoptime): New index.

* src/cuirass/database.scm: Remove unused modules.

Searching for something like "%a%" can take up to ten minutes. Remove the
usage of special characters "^" and "$" in queries. Instead, always add a "%"
character at the end of the query. Also add an index on the nix_name field.

Fixes: <https://issues.guix.gnu.org/43791>.

* src/sql/upgrade-13.sql: New file.
* Makefile.am (dist_sql_DATA): Add it.
* src/schema.sql (Builds_nix_name): New index.
* src/cuirass/database.scm (query->bind-arguments): Remove support for "^" and
"$" special characters. Instead make sure that the query does not contain any
"%" character and prefix the query by "%".
(db-get-builds-by-search): Remove an useless query nesting level.
* src/cuirass/templates.scm (search-form): Adapt the search help message
accordingly.

This is a follow-up of 761443bc.

* src/cuirass/base.scm (fetch-inputs): Use map to replace the previous par-map
call.
(compile-checkouts): Also replace par-map call by map.

Fixes: <https://issues.guix.gnu.org/43757>.

* src/cuirass/base.scm (fetch-inputs): Call par-map from a separate thread as
this call is not suspendable.

The number of database worker is limited, and can introduce some
starvation. Print a message each time a two seconds timeout expires while
sending an SQL query to a worker.

* src/cuirass/utils.scm (put-message-with-timeout): New procedure.
(call-with-worker-thread): Rename timeout and timeout-proc arguments to
receive-timeout and receive-timeout-proc arguments. Add two new arguments:
send-timeout and send-timeout-proc. Use put-message-with-timeout to honor
them.
* src/cuirass/database.scm (with-db-worker-thread): Adapt accordingly and pass
send-timeout and send-timeout-proc arguments.

* src/cuirass/base.scm (build-packages): Improve the logging message.

* src/cuirass/base.scm (build-packages): Print the id of the registered
evaluation.

* src/cuirass/database.scm (with-db-worker-thread): Print the caller name.

* src/cuirass/logging.scm (log-query): Add date.

* src/cuirass/database.scm (db-register-builds): Remove store argument and
assume that 'log and 'outputs properties are provided by the evaluation.
* src/cuirass/base.scm (build-packages): Adapt accordingly.