Reduces package metadata dump time by ~10%

Signed-off-by: Felix Fietkau <nbd@nbd.name>

Update the compiler check for ccache so we don't
end up with the wrong binaries. Right now the
compiler check will not be able to correctly
distinguish the compiler used for build
ARMv8 binaries from the one used to build
ARMv7 binaries.

Signed-off-by: Markus Gothe <markus.gothe@genexis.eu>

Link: https://github.com/openwrt/openwrt/pull/16290


Signed-off-by: John Crispin <john@phrozen.org>

- Disable compression for ccache's cached files.
- Disable the hashing of the CWD inside debug information. This
  increases the cache hits drastically.

Signed-off-by: Markus Gothe <markus.gothe@genexis.eu>

Add comments to build system makefile functions and variables to help
developers in understanding build system internals and ease the
development process.

This patch adds some documentation examples with proposed doxygen-like
syntax. Hopefully, this would start the discussion and result in
generation of the makefile documentation guidelines.

Signed-off-by: Sergey Matsievskiy <matsievskiysv@gmail.com>
Link: https://github.com/openwrt/openwrt/pull/16888


Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>

Permit to overwrite PACKAGE_DIR and PACKAGE_DIR_ALL variables in
rules.mk.

This is to handle a special case with the ImageBuilder where these
variable are overwrite.

The main problem is that any include calling rules.mk again (example
image.mk) will set these variables again dropping the modified value.

To keep the modified value, set the PACKAGE_DIR and PACKAGE_DIR_ALL only
if not already set. This permits the ImageBuilder to use custom
directory instead of the default one defined in rules.mk.

Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>

Usage of g++ should mimic the usage of gcc,
otherwise, part of a binary or library may have
optimizations and part of it may not,
unbeknownst to the users or developers working on a build.
This can lead to some features like, for example, FORTIFY_SOURCE,
to have less of an effect or even cause a build error on some hosts.

Therefore, let HOST_CXXFLAGS default to HOST_CFLAGS.

Fixes: 87d489f6 ("build: add HOST_CXXFLAGS for host build")
Signed-off-by: Michael Pratt <mcpratt@pm.me>
Link: https://github.com/openwrt/openwrt/pull/15853


Signed-off-by: Robert Marko <robimarko@gmail.com>

Several GNU tools such as tar, coreutils, and findutils
now build with support for 64-bit time by default
and otherwise require reconfiguring with a flag
--disable-year2038 in order to build without 64-bit time.

Some standard C libraries, for example,
certain older versions of glibc such as 2.31
have large file support but not long time bits support:

  checking for ... option to enable large file support... -D_FILE_OFFSET_BITS=64
  checking for ... option for timestamps after 2038... support not detected

This test using C code taken from largefile.m4 in gnulib
uses math and casting to check for overflow
with a macro and array pair that can only be defined
when 64-bit time support is present, and otherwise errors.
It is the exact same code used to test for 64-bit time
during the configure stage of building these tools,
so the results of this test before configure takes place
will always be in concordance with the results of
the test that takes place during the configure script.

Based on the test, the configure flag --disable-year2038
is added to every host tool build depending on the host system.

When the year 2038 problem finally comes around,
the effect of the test can be converted
from the toggling of a configure option into a build prerequisite,
requiring it to pass in order to continue building.

Signed-off-by: Michael Pratt <mcpratt@pm.me>
Link: https://github.com/openwrt/openwrt/pull/15799


Signed-off-by: Robert Marko <robimarko@gmail.com>

A new option called `USE_APK` is added which generated APK packages
(.apk) instead of OPKG packages (.ipk).

Some features like fstools `snapshot` command are not yet ported

Signed-off-by: Paul Spooren <mail@aparcar.org>

Currently when external toolchain is used, a info.mk is created (with
to-be-filled values) but is never actually filled with real values and
is never actually used in rules.mk.

This info.mk is used down the code with special packages like libgcc to
reference info like gcc version and package special library.

To mimic what is done with internal toolchain, add the include in
rules.mk for info.mk also for external library and in fix
toolchain/wrapper to fill the staging_dir info.mk.

The logic is to check if the external toolchain provide an info.mk and
if it doesn't we at least fill the GCC_VERSION with the value set in the
.config file.

With this special library like libgcc correctly reference and have set the
GCC_VERSION variable.

Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>

Make toolchain dirs define more consistent between internal and external
toolchains.

Make use of specific dirs also for intenral toolchain and generilize
include and lib inclusion.

Also set TOOLCHAIN_ROOT_DIR for internal toolchain as this is what
packages should use to reference staging toolchain directory.

Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>

The current minimum OS requirement for OpenWrt is Ubuntu 18.04, which
includes 7 and 8. 8 is necessary for ccache.

gcc and g+++ are now symlinked to staging_dir, similar to Python.

Signed-off-by: Rosen Penev <rosenp@gmail.com>

sstrip only has one functional arg. Make that a bool option, which can
easily depend on other knobs then.

This is required to be disabled for the mold linker.

Signed-off-by: Andre Heider <a.heider@gmail.com>

This explicitely adds the default linker to the target LDFLAGS.

No functional change intended.

Signed-off-by: Andre Heider <a.heider@gmail.com>

Add a variable that stores the original value of $PATH
in the host system's shell, before Make alters it.

This can be useful for when it is necessary
to ignore symlinks and programs made by the build system.

Define this new variable before all instances of
'export PATH:=' or similar.

Signed-off-by: Michael Pratt <mcpratt@pm.me>

By having a local copy of gnulib, we can:
import the latest macro fixes into any package,
get rid of some statically stored macros that were otherwise missing,
bootstrap GNU tools with the latest relevant source
without having to wait for a release or rely on git submodules,
and possibly more...

The patch assists in bootstrapping by ignoring
the building of po files using gettext,
and also to allow a user-defined path to a program
to include parameters.

Signed-off-by: Michael Pratt <mcpratt@pm.me>

Some uses cases, like with autotools, need a path for 'true'
if we use them to force something to skip.

This will work by default on Linux hosts,
and require MacOS hosts to get coreutils,
which is currently installed in the CI
for 'macos-latest' host.

In the future, prereq stage can be reworked
to search for the actual binary
instead of relying on env.

Signed-off-by: Michael Pratt <mcpratt@pm.me>

Some packages using Libtool have the possibility
of using this environment variable if set,
instead of the direct command for "file" using PATH,
or the need to patch a hard-coded path.

This is a new feature of Libtool 2.4.7

Ref: bf261073 ("tools/libtool: bump to 2.4.7")
Signed-off-by: Michael Pratt <mcpratt@pm.me>

These wrappers are not needed as CC doesn't need to be a single word.

a53b084e which introduced the wrappers
doesn't explain why they were really needed and why only for the target
and not for the host.

Moreover, name of the wrappers breaks a ccache assumption: since
v4.0-3-g6a92b4cd3a67 it has special handling for "chained" invocation
such as "ccache ccache gcc" where it skips all the "ccache*" names in
the middle and proceeds to run as if it was started as "ccache
gcc"[1][2].

This becomes important when a build system sees ccache in the PATH and
automatically enables it by prepending to CC. An example of such a
system would be autosetup as used by jimtcl. With the wrappers it breaks
as the command line ends up being just "ccache -Os..." because
"ccache_cc" gets skipped as it starts with "ccache".

[1] https://github.com/ccache/ccache/blob/master/src/ccache.cpp#L2105
[2] https://github.com/ccache/ccache/blob/master/src/Util.cpp#L802



Reported-by: Karl Palsson <karlp@etactica.com>
Signed-off-by: Paul Fertser <fercerpav@gmail.com>

STAGING_DIR may be provided from command line. We currently hardcoded
STAGING_DIR_HOST and STAGING_DIR_HOSTPKG to the default location but we
currently have some relocatable patch that derive the path from
STAGING_DIR.

Fix this and correctly derive STAGING_DIR_HOST and STAGING_DIR_HOSTPKG
from STAGING_DIR.

The intention is to fix inconsistency from the relocatable patch and the
use of STAGING_DIR_HOST that is always hardcoded.
This with a wrong configuration may end up in broken state with some
host tools expecing a PATH from STAGING_DIR and others using library
from the default staging_dir/host path.

To save downstream project the original implementation is saved while
fixing the inconsistency between patch and .mk.

Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>

We currently enable -Wno-error=unused-but-set-variable and
-Wno-error=unused-result by default on every compile package.

While this is (relatively) unharmful, we should follow other project
direction and starts enforcing good code quality. For example the linux
kernel recently started to enforce Wall by default and clean code is
mandatory for inclusion.

Drop for good these flags and and make it mandatory to correctly handle
return values at least with a warning log if they are not strictly error
condition.

Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>

To generate commitcount we use grep --max-count. This is not present on
alpine grep and cause wrong generation. Use -m as it's just the short
version of --max-count and more portable.

Fixes: #11200
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>

Move DOWNLOAD_CHECK_CERTIFICATE to include/download.mk as it's a better
place than exporting it in the global rules.mk makefile.

Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>

Some packages download a single raw file or set of raw files
instead of an archive.

The filenames of these download files have names and version numbers
that do not directly correspond to the name and version of the package
as it is defined in its Makefile.

To better organize downloaded files, this adds the option of
creating a subdirectory in the download directory for these files
that can be named with package specific variables.

DL_DIR is now a recursively expanded variable
since the value now depends on whether DL_SUBDIR is defined.

Signed-off-by: Michael Pratt <mcpratt@pm.me>

Remove flags from wget and curl instructing them to ignore bad server
certificates. Although other mechanisms can protect against malicious
modifications of downloads, other vectors of attack may be available
to an adversary.

TLS certificate verification can be disabled by turning oof the
"Enable TLS certificate verification during package download" option
enabled by default in the "Global build settings" in "make menuconfig"

Signed-off-by: Josh Roys <roysjosh@gmail.com>
[ add additional info on how to disable this option ]
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>

Don't add wrapped bin to the TARGET_PATH as it does cause compilation
error.

cmake.mk will use the "command -v" and will use the wrapped bin instead
of the external toolchain bin as they have the same name and command
will select the first result.

Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>

When using the OpenWrt toolchain as an external toolchain the build
failed due to missing LTO support. By choosing the GCC wrappers of
the tools this commit makes sure that the LTO-enabled executables
are being used.

Signed-off-by: Vincent Wiemann <vincent.wiemann@ironai.com>
[ wrap the commit description to 72 char ]
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>

Compiler option -no-plt will break kernel builds on some architectures
eg. (x86)  Filter this option from the recently introduced handling of
KCFLAGS vs EXTRA_OPTIMISATION

Fixes: 1d42af72 ("kernel: use KCFLAGS for passing EXTRA_OPTIMIZATION flags")

Suggested-by: Felix Fietkau <nbd@nbd.name>

Signed-off-by: Kevin Darbyshire-Bryant <ldir@darbyshire-bryant.me.uk>

On recent macOS, /usr/bin/python3 is a wrapper that finds the right python executable
It checks argv[0] to determine if python2 or python3 should be called. Always execute
it as python3 to ensure it calls the right version

Signed-off-by: Felix Fietkau <nbd@nbd.name>

Added HOST_CXXFLAGS to specify CXXFLAGS during host-compile
(e.g. to specify c++ standard: HOST_CXXFLAGS += -std=c++11)

Signed-off-by: Sergey V. Lobanov <sergey@lobanov.in>

The development branch is now on version 10, we shouldn't drag to many
old versions and therefore drop at least 7.x.

Signed-off-by: Paul Spooren <mail@aparcar.org>
Acked-by: Rosen Penev <rosenp@gmail.com>
Acked-by: Rui Salvaterra <rsalvaterra@gmail.com>
Acked-by: Hauke Mehrtens <hauke@hauke-m.de>
Acked-by: Aleksander Jan Bajkowski <A.Bajkowski@stud.elka.pw.edu.pl>

ninja is faster at building cmake packages than make, and according to reports
also more reliable at handling parallel builds
This commit includes a patch that adds GNU make jobserver support, in order to
allow more precise control over the number of parallel tasks

Enable parallel build by default for packages using ninja

Signed-off-by: Felix Fietkau <nbd@nbd.name>
Signed-off-by: Rosen Penev <rosenp@gmail.com>

Before this commit, it was assumed that mkhash is in the PATH. While
this was fine for the normal build workflow, this led to some issues if

    make TOPDIR="$(pwd)" -C "$pkgdir" compile

was called manually. In most of the cases, I just saw warnings like this:

    make: Entering directory '/home/.../package/gluon-status-page'
    bash: line 1: mkhash: command not found
    bash: line 1: mkhash: command not found
    bash: line 1: mkhash: command not found
    bash: line 1: mkhash: command not found
    bash: line 1: mkhash: command not found
    bash: line 1: mkhash: command not found
    bash: line 1: mkhash: command not found
    bash: line 1: mkhash: command not found
    [...]

While these were only warnings and the package still compiled sucessfully,
I also observed that some package even fail to build because of this.

After applying this commit, the variable $(MKHASH) is introduced. This
variable points to $(STAGING_DIR_HOST)/bin/mkhash, which is always the
correct path.

Signed-off-by: Leonardo Mörlein <me@irrelefant.net>

Fixes issue openwrt/packages#14921, whereby inline ASM wasn't getting
built as PIC; look at gmp-6.2.1/mpn/x86/pentium/popcount.asm for
example:

ifdef(`PIC',`
...

for a routine that exists in both PIC and non-PIC versions.

Make sure that wherever $(FPIC) gets passed as a variable expansion
that it gets quoted where necessary (such as setting environment
variables in shell commands).

Signed-off-by: Philip Prindeville <philipp@redfish-solutions.com>

This reverts commit b12288fa.
The patchelf approach is too fragile, and the only users of this have been
converted to make patching unnecessary
Leave the abi_version_str variable in place in rules.mk

Signed-off-by: Felix Fietkau <nbd@nbd.name>

This makes it possible to declare a package ABI_VERSION independent from the
upstream soname by setting PKG_ABI_VERSION in the package makefile.
The library filename is fixed up for files installed to packages and to the
staging dir. References to the original from executables within the same
package are also fixed up

Signed-off-by: Felix Fietkau <nbd@nbd.name>

The license folder is a core part of OpenWrt and all GPL-2.0 licensed.
Use SPDX license tags to allow machines to check licenses.

Signed-off-by: Paul Spooren <mail@aparcar.org>
[rebase, keep some Copyright lines, sharpen commit message]
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>

Packages that are in-tree only often lack a PKG_VERSION and only use the
PKG_RELEASE to mark changes. Using COMMITCOUNT/AUTORELEASE variables
causes an issue as both variables are empty during the metadata DUMP
phase.

Instead of leaving these variables empty and causing an error message
like below, set the variables to 0 during dumping. On actual building
the variable is evaluated causing in a value above 0.

ERROR: please fix package/utils/px5g-wolfssl/Makefile - \
	see logs/package/utils/px5g-wolfssl/dump.txt for details

Makefile:48: *** Package/px5g-wolfssl is missing the VERSION field.  Stop.

Reported-by: Daniel Golle <daniel@makrotopia.org>
Reported-by: Stijn Segers <foss@volatilesystems.org>
Reported-by: Stijn Tintel <stijn@linux-ipv6.be>
Signed-off-by: Paul Spooren <mail@aparcar.org>

The newly added $(COMMITCOUNT) variable was wrongly increased by plus
one. The addition should have been only added to $(AUTORELEASE) as
OpenWrt traditionally starts counting at one rather than zero.

$(AUTORELEASE) counts the commits since the last bump, which is zero on
the version bump commit itself.

This commit increases $(AUTORELEASE) by one while leaving $(COMMITCOUNT)
as is.

The base-files package is the only package using $(COMMITCOUNT) so far
and requires a pseudo commit to keep the PKG_RELEASE correct. A non
functional change (Copyright bump) is done in the next commit.

Signed-off-by: Paul Spooren <mail@aparcar.org>

The lack of bumped PKG_RELEASE variables is a recurring theme on the
mailing list and in GitHub comments. This costs precious review time,
a rare good within the OpenWrt project.

Instead of relying on a manually set PKG_RELEASE this commit adds a
`commitcount` function that uses the number of Git commits to determine
the release. The function is called via the variables `$(AUTORELEASE)`
or `$(COMMITCOUNT)`. The `PKG_RELEASE` variable can be set to either of
the two.

- $(AUTORELEASE):

Release is automagically set to the number of commits since the last
commit containing either ": update to " or ": bump to ".

Example below:

    $ git log packages/foobar/
    foobar: fixup file location
    foobar: disable docs
    foobar: bump to 5.3.2
    foobar: fixup copyright

Resulting package name: foobar_5.3.2-3_all.ipk, two package changes
since the last upstream version change, using a 1 based counter.

- $(COMMITCOUNT):

For non-traditional versioning (x.y.z), most prominent `base-files`,
this variable contains the total number of package commits.

The new functionality can also be used by other feeds like packages.git.

In case no build information is available, e.g. when using release
tarballs, the SOURCE_DATE_EPOCH is used to have a reproducible release
identifier.

Suggested-by: Daniel Golle <daniel@makrotopia.org>
Signed-off-by: Paul Spooren <mail@aparcar.org>

Some packages fail to build on arm64 when PKG_ASLR_PIE_ALL=y, due to
machine-specific size restrictions on the global offset table.
While the manual instructs to recompile with -fPIC if it fails with
-fpic, by doing this per package, there is still a risk of random
breakage due to version bumps or other changes, so let's use -fPIC on
arm64 by default.

While comparing the sizes of 141 packages built with -fpic vs -fPIC,
most packages are either equal or smaller in size. Only 9 of the
compared packages turned out slightly larger.

Signed-off-by: Stijn Tintel <stijn@linux-ipv6.be>