Cleanup PIP package tools and installation documentation.

There are a few users to consider:

1. A developer on TFF wants source at HEAD.
2. A researcher using TFF wants pip package at release in 90% of scenarios and sometimes (ideally rarely) wants nightly, but never wants to build a pip package.
3. A developer making a framework using TFF wants pip release and nightly, but never wants to build a pip package.
4. A researcher upstreaming change to TFF basically has moved from #2 to #3.

Fixes: #878
PiperOrigin-RevId: 321226038

docs/install.md

@@ -13,7 +13,7 @@ There are a few ways to set up your environment to use TensorFlow Federated
     [build TensorFlow Federated](#build-the-tensorflow-federated-pip-package)
     from source.
 
-## Install TensorFlow Federated using pip
+## Install TensorFlow Federated using `pip`
 
 #### 1. Install the Python development environment.
 
@@ -22,7 +22,7 @@ On Ubuntu:
 <pre class="prettyprint lang-bsh">
 <code class="devsite-terminal">sudo apt update</code>
 <code class="devsite-terminal">sudo apt install python3-dev python3-pip  # Python 3</code>
-<code class="devsite-terminal">sudo pip3 install --upgrade virtualenv  # system-wide install</code>
+<code class="devsite-terminal">sudo pip3 install --user --upgrade virtualenv</code>
 </pre>
 
 On macOS:
@@ -32,7 +32,7 @@ On macOS:
 <code class="devsite-terminal">export PATH="/usr/local/bin:/usr/local/sbin:$PATH"</code>
 <code class="devsite-terminal">brew update</code>
 <code class="devsite-terminal">brew install python  # Python 3</code>
-<code class="devsite-terminal">sudo pip3 install --upgrade virtualenv  # system-wide install</code>
+<code class="devsite-terminal">sudo pip3 install --user --upgrade virtualenv</code>
 </pre>
 
 #### 2. Create a virtual environment.
@@ -45,30 +45,39 @@ On macOS:
 
 Note: To exit the virtual environment, run `deactivate`.
 
-#### 3. Install the TensorFlow Federated `pip` package.
+#### 3. Install the TensorFlow Federated Python package.
 
 <pre class="prettyprint lang-bsh">
 <code class="devsite-terminal tfo-terminal-venv">pip install --upgrade tensorflow_federated</code>
 </pre>
 
-#### 4. (Optional) Test Tensorflow Federated.
+#### 4. Test Tensorflow Federated.
 
 <pre class="prettyprint lang-bsh">
 <code class="devsite-terminal tfo-terminal-venv">python -c "import tensorflow_federated as tff; print(tff.federated_computation(lambda: 'Hello World')())"</code>
 </pre>
 
-Success: TensorFlow Federated is now installed.
+Success: The latest TensorFlow Federated Python package is now installed.
 
-## Build the TensorFlow Federated pip package
+## Build the TensorFlow Federated Python package from source
 
-### 1. Install the Python development environment.
+Building a TensorFlow Federated Python package from source is helpful when you
+want to:
+
+*   Make changes to TensorFlow Federated and test those changes in a component
+    that uses TensorFlow Federated before those changes are submitted or
+    released.
+*   Use changes that have been submitted to TensorFlow Federated but have not
+    been released.
+
+#### 1. Install the Python development environment.
 
 On Ubuntu:
 
 <pre class="prettyprint lang-bsh">
 <code class="devsite-terminal">sudo apt update</code>
 <code class="devsite-terminal">sudo apt install python3-dev python3-pip  # Python 3</code>
-<code class="devsite-terminal">sudo pip3 install --upgrade virtualenv  # system-wide install</code>
+<code class="devsite-terminal">sudo pip3 install --user --upgrade virtualenv</code>
 </pre>
 
 On macOS:
@@ -78,57 +87,40 @@ On macOS:
 <code class="devsite-terminal">export PATH="/usr/local/bin:/usr/local/sbin:$PATH"</code>
 <code class="devsite-terminal">brew update</code>
 <code class="devsite-terminal">brew install python  # Python 3</code>
-<code class="devsite-terminal">sudo pip3 install --upgrade virtualenv  # system-wide install</code>
+<code class="devsite-terminal">sudo pip3 install --user --upgrade virtualenv</code>
 </pre>
 
-### 2. Install Bazel.
+#### 2. Install Bazel.
 
 [Install Bazel](https://docs.bazel.build/versions/master/install.html), the
 build tool used to compile Tensorflow Federated.
 
-### 3. Clone the Tensorflow Federated repository.
+#### 3. Clone the Tensorflow Federated repository.
 
 <pre class="prettyprint lang-bsh">
 <code class="devsite-terminal">git clone https://github.com/tensorflow/federated.git</code>
 <code class="devsite-terminal">cd "federated"</code>
 </pre>
 
-### 4. Create a virtual environment.
-
-<pre class="prettyprint lang-bsh">
-<code class="devsite-terminal">virtualenv --python python3 "venv"</code>
-<code class="devsite-terminal">source "venv/bin/activate"</code>
-<code class="devsite-terminal tfo-terminal-venv">pip install --upgrade pip</code>
-</pre>
-
-Note: To exit the virtual environment, run `deactivate`.
-
-### 5. Install Tensorflow Federated dependencies.
-
-<pre class="prettyprint lang-bsh">
-<code class="devsite-terminal tfo-terminal-venv">pip install --requirement "requirements.txt"</code>
-</pre>
-
-### 6. (Optional) Test Tensorflow Federated.
-
-<pre class="prettyprint lang-bsh">
-<code class="devsite-terminal tfo-terminal-venv">bazel test //tensorflow_federated/...</code>
-</pre>
-
-#### 7. Build the pip package.
+#### 4. Build the TensorFlow Federated Python package.
 
 <pre class="prettyprint lang-bsh">
 <code class="devsite-terminal">mkdir "/tmp/tensorflow_federated"</code>
 <code class="devsite-terminal">bazel run //tensorflow_federated/tools/development:build_pip_package -- \
-    "/tmp/tensorflow_federated"</code>
+    --nightly \
+    --output_dir "/tmp/tensorflow_federated"</code>
 </pre>
 
-#### 8. Create a new project.
+#### 5. Create a new project.
 
 <pre class="prettyprint lang-bsh">
 <code class="devsite-terminal">mkdir "/tmp/project"</code>
 <code class="devsite-terminal">cd "/tmp/project"</code>
+</pre>
 
+#### 6. Create a virtual environment.
+
+<pre class="prettyprint lang-bsh">
 <code class="devsite-terminal">virtualenv --python python3 "venv"</code>
 <code class="devsite-terminal">source "venv/bin/activate"</code>
 <code class="devsite-terminal tfo-terminal-venv">pip install --upgrade pip</code>
@@ -136,60 +128,17 @@ Note: To exit the virtual environment, run `deactivate`.
 
 Note: To exit the virtual environment run `deactivate`.
 
-#### 9. Install the pip package.
+#### 7. Install the TensorFlow Federated Python package.
 
 <pre class="prettyprint lang-bsh">
-<code class="devsite-terminal tfo-terminal-venv">pip install --upgrade "/tmp/tensorflow_federated/tensorflow_federated-"*".whl"</code>
+<code class="devsite-terminal tfo-terminal-venv">pip install --upgrade "/tmp/tensorflow_federated/"*".whl"</code>
 </pre>
 
-#### 10. Test Tensorflow Federated.
+#### 8. Test Tensorflow Federated.
 
 <pre class="prettyprint lang-bsh">
 <code class="devsite-terminal tfo-terminal-venv">python -c "import tensorflow_federated as tff; print(tff.federated_computation(lambda: 'Hello World')())"</code>
 </pre>
 
-Success: The TensorFlow Federated package is built.
-
-## Using Docker
-
-Create a Tensorflow Federated development environment using Docker on Ubuntu or
-macOS.
-
-### 1. Install Docker.
-
-[Install Docker](https://docs.docker.com/install/) on your local machine.
-
-### 2. Clone the latest Tensorflow Federated source.
-
-<pre class="prettyprint lang-bsh">
-<code class="devsite-terminal">git clone https://github.com/tensorflow/federated.git</code>
-<code class="devsite-terminal">cd "federated"</code>
-</pre>
-
-### 3. Build a Docker image.
-
-<pre class="prettyprint lang-bsh">
-<code class="devsite-terminal">docker build . \
-    --tag tensorflow_federated</code>
-</pre>
-
-### 4. Start a Docker container.
-
-<pre class="prettyprint lang-bsh">
-<code class="devsite-terminal">docker run \
-    --interactive \
-    --tty \
-    --volume $(pwd):/federated \
-    --workdir /federated \
-    tensorflow_federated \
-    bash</code>
-</pre>
-
-### 5. (Optional) Test Tensorflow Federated.
-
-<pre class="prettyprint lang-bsh">
-<code class="devsite-terminal">bazel test //tensorflow_federated/...</code>
-</pre>
-
-Success: The TensorFlow Federated development environment is ready, now
-[build the pip package](#build-the-pip-package).
+Success: A TensorFlow Federated Python package is now built from source and
+installed.

tensorflow_federated/tools/client/build_image.sh

@@ -25,6 +25,7 @@
 set -e
 
 main() {
+  # Parse arguments
   local artifacts_dir="$1"
 
   if [[ -z "${artifacts_dir}" ]]; then
@@ -32,12 +33,13 @@ main() {
     trap "rm -rf ${artifacts_dir}" EXIT
   fi
 
+  # Create working directory
   cp -LR "tensorflow_federated" "${artifacts_dir}"
   pushd "${artifacts_dir}"
 
   # Build the TensorFlow Federated package
   tensorflow_federated/tools/development/build_pip_package \
-      "${artifacts_dir}"
+      --output_dir "${artifacts_dir}"
 
   # Build the TensorFlow Federated runtime image
   docker build \

tensorflow_federated/tools/development/build_pip_package.sh

@@ -17,11 +17,15 @@
 #
 # Usage:
 #   bazel run //tensorflow_federated/tools/development:build_pip_package -- \
-#       "/tmp/tensorflow_federated"
+#       --output_dir "/tmp/tensorflow_federated"
+#   bazel run //tensorflow_federated/tools/development:build_pip_package -- \
+#       --nightly \
+#       --output_dir "/tmp/tensorflow_federated"
 #
 # Arguments:
+#   nightly: A flag indicating whether or not to build the nightly version of
+#     the pip package.
 #   output_dir: An output directory.
-#   project_name: A project name, defaults to `tensorflow_federated`.
 set -e
 
 die() {
@@ -29,18 +33,46 @@ die() {
   exit 1
 }
 
+usage() {
+  local script_name=$(basename "${0}")
+  echo "usage: ${script_name} [--nightly] [--output_dir PATH]" 1>&2
+}
+
 main() {
-  local output_dir="$1"
-  local project_name="$2"
+  # Parse arguments
+  local nightly=0
+  local output_dir=""
 
-  if [[ ! -d "${output_dir}" ]]; then
-    die "The output directory '${output_dir}' does not exist."
+  while [[ "$#" -gt 0 ]]; do
+    opt="$1"
+    case "${opt}" in
+      --nightly)
+        nightly="1"
+        shift
+        ;;
+      --output_dir)
+        output_dir="$2"
+        shift
+        # Shift might exit with an error code if no output_dir was provided.
+        shift || break
+        ;;
+      *)
+        usage
+        exit 1
+        ;;
+    esac
+  done
+
+  if [[ -z ${output_dir} ]]; then
+    usage
+    exit 1
   fi
 
-  if [[ -z "${project_name}" ]]; then
-    project_name="tensorflow_federated"
+  if [[ ! -d "${output_dir}" ]]; then
+    die "The output directory '${output_dir}' does not exist."
   fi
 
+  # Create working directory
   local temp_dir="$(mktemp -d)"
   trap "rm -rf ${temp_dir}" EXIT
   cp -LR "tensorflow_federated" "${temp_dir}"
@@ -52,10 +84,14 @@ main() {
   pip install --upgrade pip
 
   # Build pip package
+  flags=()
+  if [[ ${nightly} == "1" ]]; then
+    flags+=("--nightly")
+  fi
   pip install --upgrade setuptools wheel
   python "tensorflow_federated/tools/development/setup.py" bdist_wheel \
       --universal \
-      --project_name "${project_name}"
+      "${flags[@]}"
   popd
 
   cp "${temp_dir}/dist/"* "${output_dir}"

tensorflow_federated/tools/development/publish_pip_package.sh

@@ -17,7 +17,7 @@
 #
 # Usage:
 #   bazel run //tensorflow_federated/tools/development:publish_pip_package -- \
-#       "/tmp/tensorflow_federated/tensorflow_federated-"*".whl"
+#       --package "/tmp/tensorflow_federated/"*".whl"
 #
 # Arguments:
 #   package: A path to a local pip package.
@@ -28,13 +28,41 @@ die() {
   exit 1
 }
 
+usage() {
+  local script_name=$(basename "${0}")
+  echo "usage: ${script_name} [--package PATH]" 1>&2
+}
+
 main() {
-  local package="$1"
+  # Parse arguments
+  local package=""
+
+  while [[ "$#" -gt 0 ]]; do
+    opt="$1"
+    case "${opt}" in
+      --package)
+        package="$2"
+        shift
+        # Shift might exit with an error code if no output_dir was provided.
+        shift || break
+        ;;
+      *)
+        usage
+        exit 1
+        ;;
+    esac
+  done
+
+  if [[ -z ${package} ]]; then
+    usage
+    exit 1
+  fi
 
   if [[ ! -f "${package}" ]]; then
     die "The package '${package}' does not exist."
   fi
 
+  # Create working directory
   local temp_dir="$(mktemp -d)"
   trap "rm -rf ${temp_dir}" EXIT
   pushd "${temp_dir}"

tensorflow_federated/tools/development/setup.py

@@ -54,19 +54,6 @@ import setuptools
 
 DOCLINES = __doc__.split('\n')
 
-project_name = 'tensorflow_federated'
-
-if '--project_name' in sys.argv:
-  project_name_idx = sys.argv.index('--project_name')
-  project_name = sys.argv[project_name_idx + 1]
-  sys.argv.remove('--project_name')
-  sys.argv.pop(project_name_idx)
-
-with open('tensorflow_federated/version.py') as fp:
-  globals_dict = {}
-  exec(fp.read(), globals_dict)  # pylint: disable=exec-used
-  VERSION = globals_dict['__version__']
-
 REQUIRED_PACKAGES = [
     'absl-py~=0.9.0',
     'attrs~=19.3.0',
@@ -84,8 +71,25 @@ REQUIRED_PACKAGES = [
     'tensorflow~=2.2.0',
 ]
 
+if '--nightly' in sys.argv:
+  sys.argv.remove('--nightly')
+  PROJECT_NAME = 'tff_nightly'
+  for index, required_package in enumerate(REQUIRED_PACKAGES):
+    package_name = get_package_name(required_package)
+    if package_name == 'tensorflow':
+      REQUIRED_PACKAGES[index] = 'tf-nightly'
+    elif package_name == 'tensorflow-addons':
+      REQUIRED_PACKAGES[index] = 'tfa-nightly'
+else:
+  PROJECT_NAME = 'tensorflow_federated'
+
+with open('tensorflow_federated/version.py') as fp:
+  globals_dict = {}
+  exec(fp.read(), globals_dict)  # pylint: disable=exec-used
+  VERSION = globals_dict['__version__']
+
 setuptools.setup(
-    name=project_name,
+    name=PROJECT_NAME,
     version=VERSION,
     packages=setuptools.find_packages(exclude=('tools')),
     description=DOCLINES[0],
@@ -97,7 +101,7 @@ setuptools.setup(
     download_url='https://github.com/tensorflow/federated/tags',
     install_requires=REQUIRED_PACKAGES,
     # PyPI package information.
-    classifiers=(
+    classifiers=[
         'Development Status :: 5 - Production/Stable',
         'Intended Audience :: Developers',
         'Intended Audience :: Education',
@@ -112,7 +116,16 @@ setuptools.setup(
         'Topic :: Software Development',
         'Topic :: Software Development :: Libraries',
         'Topic :: Software Development :: Libraries :: Python Modules',
-    ),
+    ],
     license='Apache 2.0',
     keywords='tensorflow federated machine learning',
 )
+
+
+def get_package_name(requirement: str) -> str:
+  allowed_operators = ['~=', '<', '>', '==', '<=', '>=', '!=']
+  separator = allowed_operators[0]
+  for operator in allowed_operators[1:]:
+    requirement = requirement.replace(operator, separator)
+  name, _ = requirement.split(separator, maxsplit=1)
+  return name

tensorflow_federated/tools/development/test_pip_package.sh

@@ -17,7 +17,7 @@
 #
 # Usage:
 #   bazel run //tensorflow_federated/tools/development:test_pip_package -- \
-#       "/tmp/tensorflow_federated/tensorflow_federated-"*".whl"
+#       --package "/tmp/tensorflow_federated/"*".whl"
 #
 # Arguments:
 #   package: A path to a local pip package.
@@ -28,13 +28,37 @@ die() {
   exit 1
 }
 
+usage() {
+  local script_name=$(basename "${0}")
+  echo "usage: ${script_name} [--package PATH]" 1>&2
+}
+
 main() {
-  local package="$1"
+  # Parse arguments
+  local package=""
 
-  if [[ ! -f "${package}" ]]; then
-    die "The package '${package}' does not exist."
+  while [[ "$#" -gt 0 ]]; do
+    opt="$1"
+    case "${opt}" in
+      --package)
+        package="$2"
+        shift
+        # Shift might exit with an error code if no output_dir was provided.
+        shift || break
+        ;;
+      *)
+        usage
+        exit 1
+        ;;
+    esac
+  done
+
+  if [[ -z ${package} ]]; then
+    usage
+    exit 1
   fi
 
+  # Create working directory
   local temp_dir="$(mktemp -d)"
   trap "rm -rf ${temp_dir}" EXIT
   pushd "${temp_dir}"

tensorflow_federated/tools/runtime/container/build_image.sh

@@ -25,6 +25,7 @@
 set -e
 
 main() {
+  # Parse arguments
   local artifacts_dir="$1"
 
   if [[ -z "${artifacts_dir}" ]]; then
@@ -32,12 +33,13 @@ main() {
     trap "rm -rf ${artifacts_dir}" EXIT
   fi
 
+  # Create working directory
   cp -LR "tensorflow_federated" "${artifacts_dir}"
   pushd "${artifacts_dir}"
 
   # Build the TensorFlow Federated package
   tensorflow_federated/tools/development/build_pip_package \
-      "${artifacts_dir}"
+      --output_dir "${artifacts_dir}"
 
   # Build the TensorFlow Federated runtime image
   docker build \