Skip to content
Snippets Groups Projects
  • Jason Rudolph's avatar
    e163eb86
    Enhance test suite to emit JUnit XML test reports · e163eb86
    Jason Rudolph authored
    In preparation for detecting flaky tests with BuildPulse, this commit
    sets up the rspec_junit_formatter gem to output JUnit XML reports of the
    test suite, which is the format used by BuildPulse and various other
    tooling that interprets test results.
    
    Because the test suite uses the parallel_tests gem, this commit
    incorporates some related changes to make all the parallel_tests gem and
    the rspec_junit_formatter gem to cooperate with each other.
    
    rspec_junit_formatter writes everything to a single XML file. That works
    fine when there's only one process writing to the file. By default,
    whatever process finishes last will write to the file and clobber the
    output of all the other processes that wrote to the file. :see_no_evil:
    
    To prevent this issue, the parallel_tests wiki recommends adding a
    `.rspec_parallel` file to specify its RSpec options
    (https://github.com/grosser/parallel_tests/wiki#with-rspec_junit_formatter----by-jgarber),
    then the project can specify different files for each process to write
    to like so:
    
      --format RspecJunitFormatter
      --out tmp/rspec<%= ENV['TEST_ENV_NUMBER'] %>.xml
    
    However, prior to this commit, the Homebrew/brew test suite specified
    its RSpec options via the command line. Unfortunately though, there's no
    way (AFAICT) to set the equivalent of these options via the command
    line:
    
      --format RspecJunitFormatter
      --out tmp/rspec<%= ENV['TEST_ENV_NUMBER'] %>.xml
    
    So, we need to use a `.rspec_parallel` file to specify these options :point_up:.
    
    However, it appears that RSpec allows you to specify formatters _either_
    in an options file (like `.rspec_parallel`) _or_ via command-line args.
    But if you specify any formatters via command-line args, then all
    formatters in the options file are ignored.  (I suspect that's somehow
    related to this bit of code in rspec-core:
    https://github.com/rspec/rspec-core/blob/v3.10.0/lib/rspec/core/configuration_options.rb#L64.)
    
    With that in mind, in order to have the RspecJunitFormatter configured
     in `.rspec_parallel`, we need to move the other formatters into
     `.rpsec_parallel` as well, instead of passing them as command-line
     args. Therefore, this commit moves all the formatters into a
     `.rspec_parallel` file.
    Enhance test suite to emit JUnit XML test reports
    Jason Rudolph authored
    In preparation for detecting flaky tests with BuildPulse, this commit
    sets up the rspec_junit_formatter gem to output JUnit XML reports of the
    test suite, which is the format used by BuildPulse and various other
    tooling that interprets test results.
    
    Because the test suite uses the parallel_tests gem, this commit
    incorporates some related changes to make all the parallel_tests gem and
    the rspec_junit_formatter gem to cooperate with each other.
    
    rspec_junit_formatter writes everything to a single XML file. That works
    fine when there's only one process writing to the file. By default,
    whatever process finishes last will write to the file and clobber the
    output of all the other processes that wrote to the file. :see_no_evil:
    
    To prevent this issue, the parallel_tests wiki recommends adding a
    `.rspec_parallel` file to specify its RSpec options
    (https://github.com/grosser/parallel_tests/wiki#with-rspec_junit_formatter----by-jgarber),
    then the project can specify different files for each process to write
    to like so:
    
      --format RspecJunitFormatter
      --out tmp/rspec<%= ENV['TEST_ENV_NUMBER'] %>.xml
    
    However, prior to this commit, the Homebrew/brew test suite specified
    its RSpec options via the command line. Unfortunately though, there's no
    way (AFAICT) to set the equivalent of these options via the command
    line:
    
      --format RspecJunitFormatter
      --out tmp/rspec<%= ENV['TEST_ENV_NUMBER'] %>.xml
    
    So, we need to use a `.rspec_parallel` file to specify these options :point_up:.
    
    However, it appears that RSpec allows you to specify formatters _either_
    in an options file (like `.rspec_parallel`) _or_ via command-line args.
    But if you specify any formatters via command-line args, then all
    formatters in the options file are ignored.  (I suspect that's somehow
    related to this bit of code in rspec-core:
    https://github.com/rspec/rspec-core/blob/v3.10.0/lib/rspec/core/configuration_options.rb#L64.)
    
    With that in mind, in order to have the RspecJunitFormatter configured
     in `.rspec_parallel`, we need to move the other formatters into
     `.rpsec_parallel` as well, instead of passing them as command-line
     args. Therefore, this commit moves all the formatters into a
     `.rspec_parallel` file.
Code owners
Assign users and groups as approvers for specific file changes. Learn more.
This project manages its dependencies using Bundler. Learn more
Gemfile.lock 4.91 KiB
GEM
  remote: https://rubygems.org/
  specs:
    activesupport (6.1.3.2)
      concurrent-ruby (~> 1.0, >= 1.0.2)
      i18n (>= 1.6, < 2)
      minitest (>= 5.1)
      tzinfo (~> 2.0)
      zeitwerk (~> 2.3)
    addressable (2.7.0)
      public_suffix (>= 2.0.2, < 5.0)
    ast (2.4.2)
    bindata (2.4.10)
    bootsnap (1.7.5)
      msgpack (~> 1.0)
    byebug (11.1.3)
    coderay (1.1.3)
    colorize (0.8.1)
    commander (4.6.0)
      highline (~> 2.0.0)
    concurrent-ruby (1.1.9)
    connection_pool (2.2.5)
    diff-lcs (1.4.4)
    docile (1.4.0)
    domain_name (0.5.20190701)
      unf (>= 0.0.5, < 1.0.0)
    elftools (1.1.3)
      bindata (~> 2)
    highline (2.0.3)
    hpricot (0.8.6)
    http-cookie (1.0.4)
      domain_name (~> 0.5)
    i18n (1.8.10)
      concurrent-ruby (~> 1.0)
    mechanize (2.8.1)
      addressable (~> 2.7)
      domain_name (~> 0.5, >= 0.5.20190701)
      http-cookie (~> 1.0, >= 1.0.3)
      mime-types (~> 3.0)
      net-http-digest_auth (~> 1.4, >= 1.4.1)
      net-http-persistent (>= 2.5.2, < 5.0.dev)
      nokogiri (~> 1.11, >= 1.11.2)
      rubyntlm (~> 0.6, >= 0.6.3)
      webrick (~> 1.7)
      webrobots (~> 0.1.2)
    method_source (1.0.0)
    mime-types (3.3.1)
      mime-types-data (~> 3.2015)
    mime-types-data (3.2021.0225)
    mini_portile2 (2.5.3)
    minitest (5.14.4)
    msgpack (1.4.2)
    mustache (1.1.1)
    net-http-digest_auth (1.4.1)
    net-http-persistent (4.0.1)
      connection_pool (~> 2.2)
    nokogiri (1.11.7)
      mini_portile2 (~> 2.5.0)
      racc (~> 1.4)
    parallel (1.20.1)
    parallel_tests (3.7.0)
      parallel
    parlour (6.0.0)
      commander (~> 4.5)
      parser
      rainbow (~> 3.0)
      sorbet-runtime (>= 0.5)
    parser (3.0.1.1)
      ast (~> 2.4.1)
    patchelf (1.3.0)
      elftools (>= 1.1.3)
    plist (3.6.0)
    pry (0.14.1)
      coderay (~> 1.1)
      method_source (~> 1.0)
    public_suffix (4.0.6)
    racc (1.5.2)
    rack (2.2.3)
    rainbow (3.0.0)
    rdiscount (2.2.0.2)
    regexp_parser (2.1.1)
    rexml (3.2.5)
    ronn (0.7.3)
      hpricot (>= 0.8.2)
      mustache (>= 0.7.0)
      rdiscount (>= 1.5.8)
    rspec (3.10.0)
      rspec-core (~> 3.10.0)
      rspec-expectations (~> 3.10.0)
      rspec-mocks (~> 3.10.0)
    rspec-core (3.10.1)
      rspec-support (~> 3.10.0)
    rspec-expectations (3.10.1)
      diff-lcs (>= 1.2.0, < 2.0)
      rspec-support (~> 3.10.0)
    rspec-github (2.3.1)
      rspec-core (~> 3.0)
    rspec-its (1.3.0)
      rspec-core (>= 3.0.0)
      rspec-expectations (>= 3.0.0)
    rspec-mocks (3.10.2)
      diff-lcs (>= 1.2.0, < 2.0)
      rspec-support (~> 3.10.0)
    rspec-retry (0.6.2)
      rspec-core (> 3.3)
    rspec-sorbet (1.8.0)
      sorbet
      sorbet-runtime
    rspec-support (3.10.2)
    rspec-wait (0.0.9)
      rspec (>= 3, < 4)
    rspec_junit_formatter (0.4.1)
      rspec-core (>= 2, < 4, != 2.12.0)
    rubocop (1.17.0)
      parallel (~> 1.10)
      parser (>= 3.0.0.0)
      rainbow (>= 2.2.2, < 4.0)
      regexp_parser (>= 1.8, < 3.0)
      rexml
      rubocop-ast (>= 1.7.0, < 2.0)
      ruby-progressbar (~> 1.7)
      unicode-display_width (>= 1.4.0, < 3.0)
    rubocop-ast (1.7.0)
      parser (>= 3.0.1.1)
    rubocop-performance (1.11.3)
      rubocop (>= 1.7.0, < 2.0)
      rubocop-ast (>= 0.4.0)
    rubocop-rails (2.10.1)
      activesupport (>= 4.2.0)
      rack (>= 1.1)
      rubocop (>= 1.7.0, < 2.0)
    rubocop-rspec (2.4.0)
      rubocop (~> 1.0)
      rubocop-ast (>= 1.1.0)
    rubocop-sorbet (0.6.2)
      rubocop
    ruby-macho (2.5.1)
    ruby-progressbar (1.11.0)
    rubyntlm (0.6.3)
    simplecov (0.21.2)
      docile (~> 1.1)
      simplecov-html (~> 0.11)
      simplecov_json_formatter (~> 0.1)
    simplecov-cobertura (1.4.2)
      simplecov (~> 0.8)
    simplecov-html (0.12.3)
    simplecov_json_formatter (0.1.3)
    sorbet (0.5.6274)
      sorbet-static (= 0.5.6274)
    sorbet-runtime (0.5.6274)
    sorbet-runtime-stub (0.2.0)
    sorbet-static (0.5.6274-universal-darwin-14)
    spoom (1.0.9)
      colorize
      sorbet (~> 0.5.5)
      sorbet-runtime
      thor (>= 0.19.2)
    tapioca (0.4.23)
      bundler (>= 1.17.3)
      parlour (>= 2.1.0)
      pry (>= 0.12.2)
      sorbet-runtime
      sorbet-static (>= 0.4.4471)
      spoom
      thor (>= 0.19.2)
    thor (1.1.0)
    tzinfo (2.0.4)
      concurrent-ruby (~> 1.0)
    unf (0.1.4)
      unf_ext
    unf_ext (0.0.7.7)
    unicode-display_width (2.0.0)
    warning (1.2.0)
    webrick (1.7.0)
    webrobots (0.1.2)
    zeitwerk (2.4.2)

PLATFORMS
  ruby

DEPENDENCIES
  activesupport
  bootsnap
  byebug
  concurrent-ruby
  mechanize
  minitest
  nokogiri
  parallel_tests
  patchelf
  plist
  ronn
  rspec
  rspec-github
  rspec-its
  rspec-retry
  rspec-sorbet
  rspec-wait
  rspec_junit_formatter
  rubocop
  rubocop-ast
  rubocop-performance
  rubocop-rails
  rubocop-rspec
  rubocop-sorbet
  ruby-macho
  simplecov
  simplecov-cobertura
  sorbet
  sorbet-runtime
  sorbet-runtime-stub
  tapioca
  warning

BUNDLED WITH
   1.17.3