Previously it would be installed under cuirass/.

* Makefile.am (webmoduledir, webobjectdir): New variables.
(dist_pkgmodule_DATA): Remove src/web/server/fiberized.scm.
(dist_webmodule_DATA, nodist_webobject_DATA): New variables.
(CLEANFILES): Add $(nodist_webobject_DATA).

* src/cuirass/base.scm (clear-build-queue): New procedure.
* bin/cuirass.in (main): Call it.

* examples/random-jobs.scm (random-derivation): Add 'suffix' parameter
and honor it.
(make-random-jobs): Pass a non-empty suffix to 'random-derivation'.

* src/schema.sql: Add indices.

This reverts commit c47dfdf8.

Processing connections concurrently would require having separate
database handles.  See
<https://lists.gnu.org/archive/html/guix-devel/2018-02/msg00206.html>.

* src/cuirass/database.scm (db-add-derivation): Catch 'sqlite-error and
handle SQLITE_CONSTRAINT_PRIMARYKEY.
(SQLITE_CONSTRAINT_UNIQUE): New variable.
* tests/database.scm ("database")["db-add-derivation"]: Add extra call to
'db-add-derivation'.

Before that, 'run-server' would force sequential processing of client
requests one after another.

* src/cuirass/http.scm (run-cuirass-server): Rewrite to use its own loop
instead of 'run-server'.

* src/cuirass/utils.scm (%non-blocking): Forward exceptions to the
calling fiber.

That makes it safe from SQL injection.

* src/cuirass/database.scm (db-get-builds): Rewrite to use question
marks in SQL queries and binding through '%sqlite-exec'.
* tests/database.scm ("database")["db-get-builds"]: Exercise 'WHERE'
clauses.

The new macro automatically takes care of inserting question marks in
the SQL queries, which in turn guarantees that there are always as many
question marks and arguments.

* src/cuirass/database.scm (sqlite-exec): Rename to...
(%sqlite-exec): ... this.
(sqlite-exec/bind, sqlite-exec): New macros.
(assq-refs): Remove.
(db-add-specification): Use the new 'sqlite-exec' form.
(db-get-specifications): Correctly deal with REV or TAG being #f.
(db-add-derivation, db-get-derivation, db-add-evaluation)
(db-add-build, db-update-build-status!, db-get-outputs)
(db-get-build, db-get-stamp, db-add-stamp): Adjust to the new
'sqlite-exec' form.

* examples/random-jobs.scm (random-derivation): Tweak the computation of
SEED.  Actually use the new random state.

Previously any derivation not directly built by Cuirass would be
considered as failed because 'handle-build-event' wouldn't see any build
event.  Here we just make sure the build status recorded in the database
corresponds to reality.

* src/cuirass/base.scm (update-build-statuses!): New procedure.
(spawn-builds): Call it after 'build-derivations&'.

* src/cuirass/base.scm (build-packages)[register]: Make 'db-add-build' a
tail call.
Fix computation of 'outs'.

* examples/random-jobs.scm (make-job): Add '.' in #:job-name.

* src/cuirass/database.scm (db-update-build-status!): And "AND status != ?"
in SQL queries.
* tests/database.scm ("database")["db-update-build-status!"]: Add call
to 'db-update-build-status!'.

This is purely "defensive programming" since the default value for
'print-build-trace' is currently #t.

* src/cuirass/base.scm (with-store): Pass #:print-build-trace to
'set-build-options'.

Suggested by Danny Milosavljevic.

* src/cuirass/database.scm (sqlite-exec): Pass #:cache? to
'sqlite-prepare'.

* src/cuirass/database.scm (%sqlite-exec): Remove.
(sqlite-exec): Turn back into a procedure.  Use 'sqlite-bind'.  Add
'normalize' procedure and use it.
(db-add-specification, db-add-derivation, db-get-derivation)
(db-add-evaluation, db-add-build, db-update-build-status!)
(db-get-build, db-get-stamp, db-add-stamp): Use question marks in SQL
queries.
* src/cuirass/base.scm (build-packages)[register]: Make #:log
non-false.
* tests/database.scm (make-dummy-job): Add #:job-name, #:system,
 #:nix-name, and #:eval-id.  This is necessary because 'sqlite-bind'
would now translate #f to a real NULL (before it would translate to the
string "#f"...), and would thus report violations of the non-NULL
constraint.

Co-authored-by: Ludovic Courtès <ludo@gnu.org>

* src/cuirass/base.scm (build-packages): Remove useless 'log-message'
calls.

Reported by Danny Milosavljevic.

* src/cuirass/http.scm (url-handler) <"jobsets"> Return the result of
'db-get-specifications' as-is, not just the car.

* src/cuirass/base.scm (spawn-builds): Log the number of builds
performed, not the number of remaining builds.

* bin/cuirass.in (show-help, %options): Add "--threads".
(main): Honor it.  Pass #:parallelism to 'run-fibers'.

Works around <https://github.com/wingo/fibers/issues/19>.

The effect in practice would be that we'd usually not reach the
'close-pipe' call in 'evaluate', leaving zombie processes behind us,
never executing the continuation, and additionally spinning fast on a
sequence of epoll_wait/epoll_ctl calls.

* src/cuirass/base.scm <top level>: Monkey-patch (fibers internal).

* examples/random-jobs.scm, examples/random.scm: New files.
* Makefile.am (nobase_dist_pkgdata_DATA): Add them.

We used to have 'build-derivations' write to the custom binary port
returned by 'build-event-output-port'.  However, custom binary ports
constitute continuation barriers, thereby preventing fibers from being
suspended.

To make build log processing non-blocking, we therefore invert this
inversion of control and use a suspendable I/O procedure,
'read-line/non-blocking', when reading the build log.

* src/cuirass/base.scm (read-line/non-blocking, process-build-log)
(build-derivations&): New procedures.
(%newline, build-event-output-port): Remove.
(spawn-builds): Use 'build-derivations&' instead of 'build-derivations'
with 'build-event-output-port'.

* src/cuirass/http.scm (handle-builds-request): Wrap 'db-get-builds'
into 'non-blocking'.

* src/cuirass/logging.scm (log-monitoring-stats): New procedure.
* bin/cuirass.in (main): Add a fiber that calls it regularly.

* src/cuirass/database.scm (db-get-builds)[format-order-clause]: Add
'status+submission-time'.
* src/cuirass/http.scm (url-handler) <"queue">: Use it.

* src/cuirass/base.scm (shuffle-jobs): New procedure.
(spawn-builds): Use it.

* src/cuirass/base.scm (spawn-builds): Add 'log-message' call.

* build-aux/guix.scm (inputs): Add guile-fibers.

* examples/guix-track-git.scm (package->git-tracked): Add STORE parameter.
(guix-jobs): Pass value for STORE parameter.

* bin/cuirass.in (main): Pass #:parallel? to 'spawn-fiber' for the web
server.

* src/cuirass/utils.scm (essential-task): New procedure.
* bin/cuirass.in (main): Wrap each fiber in 'essential-task'.

* src/cuirass/http.scm (url-handler): Check whether REQUEST's method is
'GET, and return 405 if not.

This should placate 'guix-hydra-queued-builds', for instance, which
checks these values to choose between "Running" and "Scheduled".

* src/cuirass/http.scm (build->hydra-build): Provide the correct value
for #:finished and #:busy.

Suggested by Ricardo Wurmus.

* src/cuirass/database.scm (wal-mode): New procedure.
(db-open): Use it.

Fixes a bug whereby 'build-event-output-port' would sometimes read more
than COUNT from BV, which would usually result in a 'decoding-error'
exception from 'utf8->string'.

* src/cuirass/base.scm (build-event-output-port)[write!]: Use
'bytevector-range' to honor COUNT and OFFSET.

* src/cuirass/utils.scm (%weak-references): New variable.
(bytevector-range): New procedure.

Previously these calls would effectively suspend execution of all the
fibers until they had completed.

* src/cuirass/base.scm (process-specs): Wrap 'fetch-repository' and
'compile' calls in 'non-blocking'.