Homebrew-linuxbrew-core-Maintainer-Guide.md

Homebrew/linuxbrew-core Maintainer Guide

Merging formulae updates from Homebrew/homebrew-core

Linuxbrew-core is a fork of Homebrew-core and, therefore, it has to periodically merge changes made by Homebrew developers and contributors. Below we describe the steps required to merge Homebrew/homebrew-core into Linuxbrew/homebrew-core, possible conflicts and ways to resolve them. Note, that instructions below have been written for a "clean" environment and you might be able to skip some of the steps if you have done them in the past.

Preparation

First of all, we want to enable developer commands and prevent automatic updates while we do the merge:

export HOMEBREW_DEVELOPER=1
export HOMEBREW_NO_AUTO_UPDATE=1

Once we've done that, we need to get access to the merge-homebrew command that will be used for the merge. To do that we have to tap the Homebrew/linux-dev repository:

brew tap homebrew/linux-dev

Next, we have to navigate to the repository where we want to do the merge and make sure that there are 3 remotes:

• a remote named origin pointing to Linuxbrew-core,
• a remote named homebrew pointing to Homebrew-core, and
• a remote pointing to your GitHub fork of Linuxbrew-core.

Remote names origin and homebrew are hard-coded in merge-homebrew, while the remote pointing to your fork must be the same as your GitHub username, as it will be used to submit a pull request for the merge. Set the name to the $HOMEBREW_GITHUB_USER environment variable, or let hub fork add a remote for you.

brew install hub
cd $(brew --repo homebrew/core)
git remote add homebrew https://github.com/Homebrew/homebrew-core.git
hub fork --remote-name=$HOMEBREW_GITHUB_USER

Now, let's make sure that our local branch master is clean and that your fork is up-to-date with Homebrew/linuxbrew-core:

git checkout master
git fetch origin master
git reset --hard origin/master
git push --force $HOMEBREW_GITHUB_USER master

Strictly speaking, there is no need for git reset --hard origin/master and simple git merge origin/master would have been sufficient if you didn't mess with your local master branch. However, hard reset makes sure that these instructions are correct even if you did mess something up. The same is true for the --force flag for the git push command above.

By default, the following command will attempt to merge all the changes that the upstream Homebrew developers have made.

brew merge-homebrew --core

Merging all the changes from upstream in one go is usually undesirable since our build servers will time out. Instead, attempt to only merge 8-10 modified formulae.

git log --oneline master..homebrew/master will show a list of all the upstream commits since the last merge, from oldest to newest.

Pick a commit SHA-1 that will merge between 8-10 formulae (16-20 commits including bottles). Once you're satisfied with the list of updated formulae, begin the merge:

brew merge-homebrew --core --skip-style <sha>

The --skip-style argument skips running brew style, which saves time and in some cases avoids errors. The style errors can be fixed in bottle PRs later in the process when CI flags them.

Simple Conflicts

Once you issue the above command, the merge will begin and in the very end you will see the list of (conflicting) formulae that merge-homebrew could not merge automatically:

==> Conflicts
Formula/git-lfs.rb Formula/gnutls.rb Formula/godep.rb

Note, that you can also get a list of unmerged files (i.e. files with conflicts) using:

git diff --name-only --diff-filter=U

Of course, conflicts will be different every merge. You have to resolve these conflicts either manually in a text editor, or by using tools like diffuse, tkdiff, or meld, some of which are available from Homebrew. Frequently, conflicts are caused by the new versions of macOS bottles and look like:

<<<<<<< HEAD
    sha256 "bd66be269cbfe387920651c5f4f4bc01e0793034d08b5975f35f7fdfdb6c61a7" => :sierra
    sha256 "7071cb98f72c73adb30afbe049beaf947fabfeb55e9f03e0db594c568d77d69d" => :el_capitan
    sha256 "c7c0fe2464771bdcfd626fcbda9f55cb003ac1de060c51459366907edd912683" => :yosemite
    sha256 "95d4c82d38262a4bc7ef4f0a10ce2ecf90e137b67df15f8bf8df76e962e218b6" => :x86_64_linux
=======
    sha256 "ee6db42174fdc572d743e0142818b542291ca2e6ea3c20ff6a47686589cdc274" => :sierra
    sha256 "e079a92a6156e2c87c59a59887d0ae0b6450d6f3a9c1fe14838b6bc657faefaa" => :el_capitan
    sha256 "c334f91d5809d2be3982f511a3dfe9a887ef911b88b25f870558d5c7e18a15ad" => :yosemite
>>>>>>> homebrew/master

For such conflicts, simply remove the "HEAD" (Linuxbrew's) part of the conflict along with <<<<<<< HEAD, =======, and >>>>>>> homebrew/master lines. Later, we will submit a request to rebuild bottles for Linux for such formulae.

The merge-homebrew script will stage resolved conflicts for you.

Complex Conflicts

Of course, from time to time conflicts are more complicated and you have to look carefully into what's going on. An example of a slightly more complex conflict is below:

<<<<<<< HEAD
    if OS.mac?
      lib.install "out-shared/libleveldb.dylib.1.19" => "libleveldb.1.19.dylib"
      lib.install_symlink lib/"libleveldb.1.19.dylib" => "libleveldb.dylib"
      lib.install_symlink lib/"libleveldb.1.19.dylib" => "libleveldb.1.dylib"
      system "install_name_tool", "-id", "#{lib}/libleveldb.1.dylib", "#{lib}/libleveldb.1.19.dylib"
    else
      lib.install Dir["out-shared/libleveldb.so*"]
    end
=======
    lib.install "out-shared/libleveldb.dylib.1.19" => "libleveldb.1.19.dylib"
    lib.install_symlink lib/"libleveldb.1.19.dylib" => "libleveldb.dylib"
    lib.install_symlink lib/"libleveldb.1.19.dylib" => "libleveldb.1.dylib"
    MachO::Tools.change_dylib_id("#{lib}/libleveldb.1.dylib", "#{lib}/libleveldb.1.19.dylib")
>>>>>>> homebrew/master

Note, that in the "HEAD" (Linuxbrew's) part we see previous code of the Homebrew's formula wrapped in if OS.mac?. To resolve such a conflict you have to replace the contents of if OS.mac? part up until else with the contents of the bottom part of the conflict ("homebrew/master"). You also have to check if there are any obvious modifications that have to be made to the else part of the code that deals with non-macOS-related code.

Finishing the merge

Once all the conflicts have been resolved, a text editor will open with pre-populated commit message title and body:

Merge branch homebrew/master into linuxbrew/master

# Conflicts:
#       Formula/git-lfs.rb
#       Formula/gnutls.rb
#       Formula/godep.rb

Leave the title of the message unchanged and uncomment all the conflicting files. Your final commit message should be:

Merge branch homebrew/master into linuxbrew/master

Conflicts:
        Formula/git-lfs.rb
        Formula/gnutls.rb
        Formula/godep.rb

Submitting a PR

The merge-homebrew command will create a pull-request for you, using hub.

It is expected that CI checks on the merge commit of the PR will fail. This is due to a bug with Azure Pipelines and its handling of merge commits. Master branch builds also fail for the same reason. This is OK.

Once the PR is approved by other Homebrew developers, you can finalise the merge with:

brew pull --clean <PR-NUMBER>
git push origin master

The merge is now complete. Don't forget to update your GitHub fork by running git push your-fork master

Building bottles for updated formulae

After merging changes, we must rebuild bottles for all the PRs that had conflicts.

To do this, tap Homebrew/homebrew-linux-dev and run the following command where the merge commit is HEAD:

for formula in $(brew find-formulae-to-bottle); do
  brew build-bottle-pr --remote=$HOMEBREW_GITHUB_USER $formula
done

The find-formulae-to-bottle command outputs a list of formulae parsed from the merge commit body. It also performs some checks against the formulae:

And it skips formulae if any of the following are true:

• it doesn't need a bottle
• it already has a bottle
• the formula's tap is Homebrew/homebrew-core (the upstream macOS repository)
• there is already an open PR for the formula's bottle
• the current branch is not master

If a formula you are expecting to bottle is skipped, there may be an error; by default, this script won't output the errors. To see them, run brew find-formulae-to-bottle --verbose separate to the for loop above.

The build-bottle-pr script creates a branch called bottle-<FORMULA>, adds # Build a bottle for Linux to the top of the formula, pushes the branch to GitHub at the specified remote (default: origin), and opens a pull request using hub pull-request.

Pulling bottles

Pull requests are either raised by maintainers or users. In both cases, how to merge them depends on whether or not a Linux bottle has been built for the formula.

We very rarely use the GitHub UI buttons. Instead, we "pull the bottle". This means that the PR shows up as "closed" to the user, but they still get authorship credit. This is done with the following command:

HOMEBREW_BOTTLE_DOMAIN=https://linuxbrew.bintray.com brew pull --bottle --bintray-org=linuxbrew --test-bot-user=LinuxbrewTestBot <PR-NUMBER>

It saves a lot of time to alias this in your shell config. One possible alias is lbrew-pull-bottle.

For PRs with the title "Build a bottle for Linux" and that have only one commit with contents "# Build a bottle for Linux", these have been created with brew build-bottle-pr and the commit from the PR doesn't need preserving. We don't want to litter the codebase with comments. In these cases, you can combine brew pull --bottle with brew squash-bottle-pr (in the Homebrew/linux-dev tap). This will squash the first commit message, leaving just the commit with the bottle SHA authored by LinuxbrewTestBot. It will still close the PR, as brew pull --bottle adds Closes and Signed-off-by to the commit message body.

lbrew-pull-bottle <PR-NUMBER> && brew squash-bottle-pr

For PRs where there have been force pushes or extra commits to fix the build or fix bottling syntax, we can't brew squash-bottle-pr as we must keep the fixes. If the # Build a bottle for Linux line still exists in the formula, remove it.

The brew pull command publishes the bottle to BinTray and verifies that the SHA in the formula and the SHA of the downloaded file match. To verify a bottle, the script downloads the bottle from BinTray - if you're on an unstable connection, this may take a while or even time out. Publishing the bottle means that it's available as the latest version for users to download, so remember to push your commits to origin.

If something goes wrong with the bottle pull and you don't want to publish the bottle and push the commit, git reset --hard origin/master and login to BinTray and delete the new bottle (there's a list of who published what recently).

Once you've pushed to origin, there's no going back: you're a maintainer now, you can't force-push to fix your mistakes!

Creating new Linux-specific formula

Make a PR to Homebrew/linuxbrew-core containing one commit named like this: name (new formula). Keep only one commit in this PR, squash and force push to your branch if needed. Include a comment: # tag "linux" in the formula after the url stanza, so maintainers can easily find Linux only formulae. For brew pull to be successful when new formulae are added, we have to insert an empty bottle block into the formula code. This usually goes after the linux tag.

bottle do
end

Common build failures and how to handle them

Bottling errors

Handling brew bump-formula-pr PRs

Formulae that exist in Homebrew/homebrew-core

When running on Linux, the brew bump-formula-pr command should raise pull requests against the correct upstream macOS Homebrew-core repository. If a pull request is raised against the Linuxbrew-core repository when an upstream formula exists, please use the following message to direct users to the correct repository:

Thanks for your PR.

However, this formula is not Linux-specific. Its new versions are merged from the Homebrew/homebrew-core repository daily as documented in CONTRIBUTING.md. Please submit this change as a PR to that repository.

We look forward to your PR against Homebrew/homebrew-core for the next version bump!

Linux-only formulae

If the formula is a Linux-only formula, it either:

• will contain the line # tag "linux"
• won't have macOS bottles

These formulae are fine for users to bump with brew bump-formula-pr, but you should request that they remove the existing x86_64_linux bottle SHA line so that CI will build a bottle for the new version correctly. If the bottle SHA isn't removed, CI will fail with the following error:

--keep-old was passed but there are changes in sha256 => x86_64_linux