* src/cuirass/database.scm (assqx-ref): Remove exported procedure.
(db-get-builds, db-get-build): Adapt to new format.
* src/cuirass/http.scm (request-parameters): Use (cons key param) instead
of (list key param).
(url-handler): Adapt to new format.
* tests/database.scm ("db-get-builds"): Idem.

* Makefile.am (dist_sql_DATA): Add static files.
* src/cuirass/database.scm (assqx-ref): Export it.
(db-get-builds): Add 'evaluation' filter and filters for pagination.
(db-get-evaluations-build-summary, db-get-evaluations-id-min,
db-get-evaluations-id-max, db-get-builds-min, db-get-builds-max): New exported
procedures.
* src/cuirass/http.scm (%static-directory): New parameter.
(%page-size, %file-mime-types, %file-white-list): New variables.
(handle-build-request, handle-builds-request): Move the WITH-CRITICAL-SECTION
call out.
(url-handler): Add RESPOND-HTML, RESPOND-STATIC-FILE and RESPOND-NOT-FOUND
procedures.  Call WITH-CRITICAL-SECTION sooner for the '/build',
'/api/latestbuilds' and '/api/queue' routes.  Add '/', '/jobset/<name>',
'/eval/<id>', '/static/<path>' routes.  Use RESPOND-NOT-FOUND when the route
isn't found.
* src/cuirass/templates.scm: New file.
(html-page, specifications-table, evaluation-info-table,
build-eval-table): New exported procedures.
(pagination): New procedure.

Signed-off-by: Clément Lassieur <clement@lassieur.org>

This adds Bootstrap v4.1.1 (https://getbootstrap.com/) and Open Iconic
v1.1.0 (https://useiconic.com/open

).

* src/static/css/bootstrap.css: New file.
* src/static/css/open-iconic-bootstrap.css: New file.
* src/static/fonts/open-iconic.eot: New file.
* src/static/fonts/open-iconic.otf: New file.
* src/static/fonts/open-iconic.svg: New file.
* src/static/fonts/open-iconic.ttf: New file.
* src/static/fonts/open-iconic.woff: New file.
* src/static/images/logo.png: New file.

Signed-off-by: Clément Lassieur <clement@lassieur.org>

* src/cuirass/utils.scm (make-critical-section): Put the modified message to
the REPLY channel that was part of the initial message.
(call-with-critical-section): Create a REPLY channel, add it to the sent
message, get the modified message from that channel.

Co-authored-by: Ludovic Courtès <ludo@gnu.org>

This disables mutexing on database connection and prepared statement objects,
thus making us responsible for serializing access to database connections and
prepared statements.  It may result in a performance improvement.

* src/cuirass/database.scm (db-init, db-open): Pass the SQLITE_OPEN_NOMUTEX
flag to SQLITE-OPEN.

* Makefile.am (dist_sql_DATA): Add src/sql/upgrade-1.sql.
* bin/cuirass.in (show-help, %options, main): Remove the LOAD-PATH option that
was used afterwards as %GUIX-PACKAGE-PATH.
* bin/evaluate.in (absolutize, input-checkout, spec-source, spec-load-path,
spec-package-path, format-checkouts): New procedures.
(%not-colon): Remove variable.
(main): Take the load path, package path and PROC from the checkouts that
result from the inputs.  Format the checkouts before sending them to the
procedure.  Remove the LOAD-PATH argument.
* doc/cuirass.texi (Overview, Database schema): Document the changes.
* examples/{guix-jobs.scm, hello-git.scm, hello-singleton.scm,
hello-subset.scm, random.scm}: Adapt to the new specification format.
* examples/guix-track-git.scm (package->spec): Rename to PACKAGE->INPUT.
(package->git-tracked): Replace FETCH-REPOSITORY with FETCH-INPUT and handle
the new format of its return value.
* examples/random-jobs.scm (make-random-jobs): Rename RANDOM to CHECKOUT.
Rename the checkout from 'random (which is a specification) to 'cuirass (which
is a checkout resulting from an input).
* src/cuirass/base.scm (fetch-repository): Rename to fetch-input.  Rename SPEC
to INPUT.  Return a checkout object instead of returning two values.
(evaluate): Take a list of CHECKOUTS and COMMITS as arguments, instead of
SOURCE.  Remove TOKENIZE and LOAD-PATH.  Pass the CHECKOUTS instead of the
SOURCE to "evaluate".  Remove %GUIX-PACKAGE-PATH.  Build the EVAL object
instead of getting it from "evaluate".
(compile?, fetch-inputs, compile-checkouts): New procedures.
(process-specs): Fetch all inputs instead of only fetching one repository.
The result of that fetching operation is a list of CHECKOUTS whose COMMITS are
used as a STAMP.
(%guix-package-path, set-guix-package-path): Remove them.
* src/cuirass/database.scm (db-add-input, db-get-inputs): New procedures.
(db-add-specification, db-get-specifications): Adapt to the new specification
format.  Add/get all inputs as well.
(db-add-evaluation): Rename REVISION to COMMITS.  Store COMMITS as space
separated commit hashes.
(db-get-builds): Rename REPO_NAME to NAME.
(db-get-stamp): Rename COMMIT to STAMP.  Return #f when there is no STAMP.
(db-add-stamp): Rename COMMIT to STAMP.  Deal with DB-GET-STAMP's new return
value.
(db-get-evaluations): Rename REVISION to COMMITS.  Tokenize COMMITS.
* src/cuirass/utils.scm (%non-blocking): Export it.
* src/schema.sql (Inputs): New table that refers to the Specifications table.
(Specifications): Move input related fields to the Inputs table.  Rename
REPO_NAME to NAME.  Rename ARGUMENTS to PROC_ARGS.  Rename FILE to PROC_FILE.
Add LOAD_PATH_INPUTS, PACKAGE_PATH_INPUTS and PROC_INPUT fields that refer to
the Inputs table.
(Stamps): Rename REPO_NAME to NAME.
(Evaluations): Rename REPO_NAME to NAME.  Rename REVISION to COMMITS.
(Specifications_index): Replace with Inputs_index.
* src/sql/upgrade-1.sql: New file.
* tests/database.scm (example-spec, make-dummy-eval, sqlite-exec): Adapt to
the new specifications format.  Rename REVISION to COMMITS.
* tests/http.scm (evaluations-query-result, fill-db): Idem.

This removes the possibility to filter specifications by branch, because
branches were previously called 'jobset'.  But it doesn't matter because later
on, specifications will have as many branches as inputs.  And people should
filter by specification name instead.

* doc/cuirass.texi (Build Information, Latest builds): Remove 'jobset',
replace 'project' with 'jobset'.
* src/cuirass/http.scm (build->hydra-build): Idem.
* tests/database.scm (db-get-builds): Idem.
* tests/http.scm (build-query-result, /api/latestbuilds?nr=1&jobset=guix,
/api/latestbuilds?nr=1&jobset=gnu): Idem.
* src/cuirass/database.scm (db-format-build, db-get-builds): Don't associate
builds with branches (which were called 'jobset' afterwards).
(db-get-builds): Remove the #:project filter.

* src/cuirass/database.scm (%package-sql-dir): New parameter.
(db-load, db-schema-version, db-set-schema-version, latest-db-schema-version,
schema-upgrade-file, db-upgrade): New procedures.
(db-init): Set version corresponding to the existing upgrade-n.sql files.
(db-open): If database exists, upgrade it.

* src/cuirass/utils.scm (%non-blocking): Wrap body in PARAMETERIZE form that
clears CURRENT-FIBER.

So that PUT-MESSAGE doesn't try to suspend itself within CALL-WITH-NEW-THREAD.
See https://lists.gnu.org/archive/html/guile-devel/2018-07/msg00009.html.

So that Cuirass specifications used to build 'guix-modular' can be named
differently than "guix" and "guix-modular" (see Guix's
build-aux/hydra/guix-modular.scm).

The name is used as a primary key, so before that commit, it was also
impossible to have several such specifications.

This is a workaround, Cuirass should normally support several inputs per
specification, as Hydra does.  The specification's name would then only be
used to identify it.  See <https://bugs.gnu.org/31813

>.

* bin/evaluate.in (main): Replace custom NAME (passed to PROC) with 'guix'.

Co-authored-by: Mathieu Othacehe <m.othacehe@gmail.com>

* src/cuirass/database.scm (db-get-evaluations): Reverse returned evaluations
list.

* src/cuirass/database.scm (db-get-evaluations): New exported procedure.
* src/cuirass/http.scm (url-handler): Add /api/evaluations route.
* tests/http.scm ("http"): Add /api/evaluations test route.

* src/schema.sql: Extend 'Builds_Derivations_index' to account for
/api/queue queries.

* bin/cuirass.in (main): Use the 2.2 'setvbuf' API style.
* examples/guix-track-git.scm (current-error-port): Likewise.

Fixes a regression introduced in
074b9d02 whereby we'd attempt to build
all of DRV at once.

* src/cuirass/base.scm (spawn-builds): Pass BATCH, not DRV, to
'build-derivations&' and to 'update-build-statuses!'.

* bin/evaluate.in (main): Do not restore the original '%load-path'.

Previously we'd systematically add the first argument to the search
path.  When that first argument was the empty string (as with the
'guix-modular' jobset), we'd thus add the current directory to the
search path, even if the intent was to leave the load path unchanged.

Furthermore, a3a7c09b changed load-path
handling to be explicit in Scheme.

* bin/evaluate.in: Remove GUILE_LOAD_PATH assignment.

* bin/evaluate.in (with-directory-excursion): Remove.
(main): chdir to SOURCE.

This avoids a situation whereby, when evaluating from a Guix checkout,
we'd have already loaded slightly different and incompatible (guix …)
modules.

Hydra's 'hydra-eval-guile-jobs' implemented the same solution as in this
patch already.

* bin/evaluate.in: Remove use of (cuirass …) and (guix …) modules.
(ref): New procedure.
(with-directory-excursion): New macro.
(main): Use 'ref'.  Remove uses of Guix or Cuirass modules.

The previous load path was potentially incorrect since commit
2fe7ff87.

* src/cuirass/base.scm (evaluate)[tokenize, load-path]: New variables.
Assume #:load-path is colon-separated.  Pass LOAD-PATH as the second
argument to 'evaluate'.
* doc/cuirass.texi (Database): Adjust documentation.

* bin/evaluate.in (main): Prepend LOAD-PATH to '%load-path' for the
dynamic extend of the 'primitive-load' call.

Fixes a thinko introduced in
1872dd95.

* bin/evaluate.in (main): Make sure 'real-build-things' is bound.

* bin/evaluate.in (main): Use SPEC's #:name as a key in ARGS.
* examples/random-jobs.scm (make-random-jobs): Adjust accordingly.

Previously we would make a SQL query that would return many build jobs,
and then call 'delete-duplicates' on that.  This was extremely wasteful
because the list of returned by the query was huge leading to a heap of
several tens of GiB on a big database, and 'delete-duplicates' would
lead to more GC and it would take ages.

Furthermore, since 'delete-duplicates' is written in C as of Guile
2.2.3, it is uninterruptible from Fiber's viewpoint.  Consequently, the
kernel thread running the 'restart-builds' fiber would never schedule
other fibers, which could lead to deadlocks--e.g., since fibers are
scheduled on a circular shuffled list of kernel threads, once every N
times, a web server fiber would be sent to that kernel thread and not be
serviced.

* src/cuirass/base.scm (shuffle-jobs): Remove.
(shuffle-derivations): New procedure.
(spawn-builds): Take a list of derivations instead of a list of jobs.
(restart-builds): Remove 'builds' parameter.  Remove 'delete-duplicates'
call.  Remove done/remaining partitioning.
(build-packages): Adjust to pass 'spawn-builds' a list of derivations.
* bin/cuirass.in (main): Remove computation of PENDING.  Remove second
parameter in call to 'restart-builds'.

* src/cuirass/database.scm (db-get-pending-derivations): New procedure.
* tests/database.scm ("database")["db-get-pending-derivations"]: New test.

* bin/evaluate.in (main): Always pass an alist as the arguments to PROC,
containing at least 'file-name' and 'revision'.
* examples/random-jobs.scm (make-random-jobs): Display 'file-name' and
'revision' from ARGUMENTS.

* src/cuirass/utils.scm (make-critical-section)
(call-with-critical-section): New procedures.
(with-critical-section): New macro.
* src/cuirass/http.scm (with-database-access): Remove.
(handle-build-request, handle-builds-request, url-handler): Use
'with-critical-section' instead of 'with-database-access'.
(run-cuirass-server): Remove 'spawn-fiber' call.  Use
'make-critical-section' instead.

* src/cuirass/base.scm (make-writable-copy)[chmod+w]: New procedure.
Replace 'system*' call with 'file-system-fold' call.

This avoids copying things back and forth.

* src/cuirass/base.scm (fetch-repository): Add #:writable-copy?
parameter.  Call 'make-writable-copy' when it's true.
(copy-repository-cache): Remove.
(make-writable-copy): New procedure.
(evaluate): Add 'source' parameter and pass it to the 'evaluate' program.
(process-specs): Define 'compile?'.  Pass #:writable-copy? to
'fetch-repository'.  Remove call to 'copy-repository-cache'.  Remove
computation of the checkout directory name.  Pass CHECKOUT to 'evaluate'.
* bin/evaluate.in (main): Replace 'cachedir' with 'source'.  Remove
computation of the checkout directory name.

Fixes a regression introduced in
1bab5c4e whereby the
'status+submission-time' order would no longer be honored.

As a result, /api/queue would return the queue ordered by build IDs,
making it largely useless.

* src/cuirass/database.scm (db-get-builds): Remove 'order' and rename
'order-column-name' to 'order'.  Add case for 'status+submission-time'.
* tests/database.scm ("database")["db-get-builds"]: Move below
"db-update-build-status!" test.  Add case for the
'status+submission-time' order.

This reinstates c47dfdf8 and fixes the
issues that led to the revert in
b71f0cdc.

Before that, 'run-server' would force sequential processing of client
requests one after another.

* src/cuirass/http.scm (run-cuirass-server): Rewrite to use its own loop
instead of 'run-server'.  Spawn a database fiber.
(with-database-access): New macro.
(handle-build-request): Expect 'db-channel' and use 'with-database-access'.
(handle-builds-request): Likewise.
(url-handler): Likewise.

* src/cuirass/base.scm (cancel-old-builds): New procedure.

* bin/evaluate.in (main): 'build-things' replacement no longer calls 'exit'.

Fixes a bug whereby some fibers would get a SQLITE_BUSY exception while
accessing the database: see
<https://debbugs.gnu.org/cgi/bugreport.cgi?bug=30644#26>.

Suggested by Danny Milosavljevic <dannym@scratchpost.org>.

* src/cuirass/database.scm (wal-mode): Rename to...
(set-db-options): ... this.  Add call to 'sqlite-exec' for
'busy_timeout'.

* bin/cuirass.in (main): Add 'setvbuf' calls.

* src/cuirass/base.scm (with-store): Rewrite using 'unwind-protect'.
* src/cuirass/database.scm (with-database): Likewise.

* src/cuirass/utils.scm (unwind-protect): New macro.

Fixes a regression introduced in
f083282f, whereby we'd return 0 as the
timestamp for everything in /api/queue.

* src/cuirass/http.scm (build->hydra-build): Make 'finished?' a
Boolean.  Move 'bool->int' call in #:finished definition.

Previously, when an exception was raised from
'handle-build-event' (e.g., a "database is locked" error), we'd throw,
thereby leaving PORT open and we'd never read from it again.  Thus, the
corresponding 'guix-daemon' process would eventually get stuck in a
'write' call to that socket, and its build processes would stall.

* src/cuirass/base.scm (exception-reporter): New procedure.
(spawn-builds): Use it.

Fixes <https://bugs.gnu.org/30618>.
Reported by Andreas Enge <andreas@enge.fr>.

* bin/evaluate.in (fill-job): Remove.
(main): Remove 'database' command-line argument.  Remove DB and its
uses.  Write an (evaluation EVAL JOBS) sexp.
* src/cuirass/base.scm (evaluate)[augment-job]: New procedure.
Use it.  Adjust to read (evaluation EVAL JOBS) sexp.  Call
'db-add-evaluation' and 'db-add-derivation'.