[RFC] Source packages

* [RFC] Source packages
@ 2015-12-08 20:53 Haris Okanovic
  2015-12-09 17:56 ` Andre McCurdy
  2015-12-15  0:25 ` [PATCH 1/2] package.bbclass/package.py: Add do_install_source() task Haris Okanovic
  0 siblings, 2 replies; 7+ messages in thread
From: Haris Okanovic @ 2015-12-08 20:53 UTC (permalink / raw
  To: openembedded-core; +Cc: Haris Okanovic

This change enables creation of ${PN}-src packages which provide recipe
source code on target machines. A distribution might use this facility
to provide a feed of source packages for user reference and
debugging/development activities.

The packaged source code is copy of SRC_URI files + a manifest declaring
the contents in order of appearance in SRC_URI. For example, it might
include a tarball + patch files, which could be used to reconstruct the
state of a recipe's working dir after do_patch() by unpacking the
archive and applying patches in order.

At the moment, unpacking need to be done by hand. However, RPM and Dpkg
can both do this automatically upon install when the source is packaged
in a src.rpm or dsc file, respectively, as opposed to a binary archive
(rpm or deb). A good future improvement might be to provide proper
src.rpm's or .dsc's for distributions using those package manager.

I don't believe opkg can distinguish between binary and source archives
at the moment, so this may be the best we can do at the moment for IPK
distributions.


Classic OE provided a similar facility for IPKs only with
sourceipk.bbclass [1]. It was dropped at some point in modern OE and I'm
not familiar enough with OE's history pre Yocto to track down exactly
why. I'm curious if this was this a deliberate change or just an
accidental regression, and the reasons it was removed in the former
case.

One significant difference between the classic sourceipk.bbclass and my
proposal is what gets packaged: sourceipk simply archived the entire
${S} dir after do_patch() but before do_configure() pollutes it. I opted
not to go this route because it's seems at odds with RPM and Dpkg, both
of which use the aforementioned tarball+patch file scheme. This may
allow us to re-use the staging logic should we implement proper
src.rpm's and dsc's in the future.


Not all recipes have useful source to package: For example,
linux-libc-headers build from the kernel source, which is is already
provided by the kernel recipe. base-files just provides initial system
configuration, so it's source package provides no additional info than
it's binary.

I couldn't find a clever way to filter out these edge cases, other than
disabling source packages inside those recipes which would require a
lengthy review of OE meta layers.

I imagine many OE distro also target small, resource constrained,
systems, that couldn't benefit from source packaging.

Therefore I've made this an opt-in feature for individual distro
maintainers to enable as needed. It can be enabled per recipe via the
ENABLE_SRC_INSTALL_${PN} var or globally via ENABLE_SRC_INSTALL from
distribution confs.


=== Implementation details ===

[PATCH 1/3] package.bbclass/package.py: Add do_install_source() task

Add do_install_source() task to stage a recipe's SRC_URI files under
SRC_D.

Dependencies:
 After do_fetch() to ensure SRC_URI files are downloaded
 After do_install() because it resets ${D} that's also used by this task
 Before do_package() to stage files before writing installers

No-ops unless ENABLE_SRC_INSTALL_${PN} = 1, which needs to be set in
distro config or recipes wanting to use this facility.

[PATCH 2/3] documentation.conf: Document do_install_source()

[PATCH 3/3] bitbake.conf: Define source package, disabled by default

Add ${PN}-src to PACKAGES with a default set of FILES_{PV}-src
per FHS [2].

Define a staging directory SRC_D under ${D}.

Define ENABLE_SRC_INSTALL to enable source staging via package.bbclass's
do_install_source() task. It's disabled by default, but may be
overridden by distro configs or individual recipes as needed.


=== References ===

[1] https://github.com/openembedded/openembedded/blob/master/classes/sourceipk.bbclass
[2] https://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.html#usrsrcSourceCode


Comments, opinions, and other meandering thoughts welcome....

Thanks,
Haris
---
 meta/classes/package.bbclass |  5 +++
 meta/conf/bitbake.conf       |  9 ++++-
 meta/conf/documentation.conf |  1 +
 meta/lib/oe/package.py       | 96 ++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 110 insertions(+), 1 deletion(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index d731757..98f01e5 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -2079,3 +2079,8 @@ def mapping_rename_hook(d):
     runtime_mapping_rename("RRECOMMENDS", pkg, d)
     runtime_mapping_rename("RSUGGESTS", pkg, d)
 
+addtask do_install_source after do_fetch after do_install before do_package
+
+python do_install_source () {
+    oe.package.do_install_source(d)
+}
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 06971da..431865d 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -36,6 +36,7 @@ export systemd_unitdir = "${nonarch_base_libdir}/systemd"
 export systemd_system_unitdir = "${nonarch_base_libdir}/systemd/system"
 export nonarch_libdir = "${exec_prefix}/lib"
 export systemd_user_unitdir = "${nonarch_libdir}/systemd/user"
+export srcdir = "${prefix}/src"
 
 # Architecture dependent paths
 export bindir = "${exec_prefix}/bin"
@@ -267,7 +268,7 @@ SOLIBSDEV = ".so"
 SOLIBSDEV_darwin = ".dylibbroken"
 
 PACKAGE_BEFORE_PN ?= ""
-PACKAGES = "${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}"
+PACKAGES = "${PN}-src ${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}"
 PACKAGES_DYNAMIC = "^${PN}-locale-.*"
 FILES = ""
 
@@ -313,6 +314,12 @@ ALLOW_EMPTY_${PN}-dbg = "1"
 
 FILES_${PN}-locale = "${datadir}/locale"
 
+FILES_${PN}-src = "${srcdir}/${PN}"
+SRC_D = "${D}${srcdir}/${PN}"
+
+ENABLE_SRC_INSTALL = "0"
+ENABLE_SRC_INSTALL_${PN} = "${ENABLE_SRC_INSTALL}"
+
 # File manifest
 
 FILE_DIRNAME = "${@os.path.dirname(d.getVar('FILE', False))}"
diff --git a/meta/conf/documentation.conf b/meta/conf/documentation.conf
index 845559a..0df8a2f 100644
--- a/meta/conf/documentation.conf
+++ b/meta/conf/documentation.conf
@@ -26,6 +26,7 @@ do_fetchall[doc] = "Fetches all remote sources required to build a target"
 do_generate_qt_config_file[doc] = "Writes a qt.conf file for building a Qt-based application"
 do_install[doc] = "Copies files from the compilation directory to a holding area"
 do_install_ptest_base[doc] = "Copies the runtime test suite files from the compilation directory to a holding area"
+do_install_source[doc] = "Stages source code for packaging"
 do_kernel_checkout[doc] = "Checks out source/meta branches for a linux-yocto style kernel"
 do_kernel_configcheck[doc] = "Validates the kernel configuration for a linux-yocto style kernel"
 do_kernel_configme[doc] = "Assembles the kernel configuration for a linux-yocto style kernel"
diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index ea6feaa..f980f25 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -123,3 +123,99 @@ def read_shlib_providers(d):
                         shlib_provider[s[0]] = {}
                     shlib_provider[s[0]][s[1]] = (dep_pkg, s[2])
     return shlib_provider
+
+def archive_dir(dirPath, archivePath):
+    ''' Create tar.bz2 archive at archivePath from dirPath '''
+    import os, oe, bb
+
+    arDir = os.path.dirname(dirPath)
+    arName = os.path.basename(dirPath)
+
+    cmd = 'tar -c -j -f \"%s\" -C \"%s\" -p \"%s\"' % (archivePath, arDir, arName)
+    (retval, output) = oe.utils.getstatusoutput(cmd)
+    if retval:
+        bb.fatal('Failed to archive %s --> %s: %s %s' % (dirPath, archivePath, cmd, output))
+
+def do_install_source(d):
+    ''' Stage recipe's source for packaging '''
+    import os, oe, bb
+
+    pn = d.getVar("PN", True)
+
+    if d.getVar("ENABLE_SRC_INSTALL_%s" % pn, True) != "1":
+        return
+
+    packages = (d.getVar("PACKAGES") or "").split()
+    if ("%s-src" % pn) not in packages:
+        # Some recipes redefine PACKAGES without ${PN}-src. Don't stage
+        # anything in this case to avoid installed-vs-shipped warning.
+        return
+
+    urls = (d.getVar('SRC_URI', True) or "").split()
+    if len(urls) == 0:
+        return
+
+    workdir = d.getVar('WORKDIR', True)
+
+    # TODO rm_work() should clean this up
+    unpackTempDir = os.path.join(workdir, 'install-source-unpack-temp')
+    if os.path.exists(unpackTempDir):
+        bb.utils.remove(unpackTempDir, recurse=True)
+    os.makedirs(unpackTempDir, 0755)
+
+    src_d = d.getVar("SRC_D", True)
+    if os.path.exists(src_d):
+        bb.warn("SRC_D already exist. Removing.")
+        bb.utils.remove(src_d, recurse=True)
+    os.makedirs(src_d, 0755)
+
+    fetcher = bb.fetch2.Fetch(urls, d)
+
+    fileManif = []
+    for url in urls:
+        urlScheme = bb.fetch2.decodeurl(url)[0]
+        srcPath = fetcher.localpath(url)
+        srcName = os.path.basename(srcPath)
+
+        dstName = srcName
+        if os.path.isdir(srcPath):
+            dstName += '.tar.bz2'
+
+        dstPath = os.path.join(src_d, dstName)
+
+        # fetch() doesn't retrieve any actual files from git:// URLs,
+        # so we do an additional unpack() step to get something useful
+        # for these.
+        # TODO: May need to pre-process other revision control schemes
+        if urlScheme == 'git':
+            unpackPath = os.path.join(unpackTempDir, srcName)
+            if os.path.exists(unpackPath):
+                bb.utils.remove(unpackPath, recurse=True)
+            os.makedirs(unpackPath, 0755)
+
+            fetcher.unpack(unpackPath, [url])
+
+            # unpack() puts actual source in a 'git' subdir
+            srcPath = os.path.join(unpackPath, 'git')
+
+        if os.path.exists(dstPath):
+            bb.warn('Duplicate file %s in SRC_URI. Overwriting.' % dstName)
+            bb.utils.remove(dstPath, recurse=True)
+
+        if not dstName in fileManif:
+            fileManif.append(dstName)
+
+        if os.path.isdir(srcPath):
+            archive_dir(srcPath, dstPath)
+        else:
+            bb.utils.copyfile(srcPath, dstPath)
+
+    manifFilePath = os.path.join(src_d, 'manifest')
+    if os.path.exists(manifFilePath):
+        bb.warn('manifest file found in SRC_URI. Overwriting.')
+        bb.utils.remove(manifFilePath, recurse=True)
+
+    with open(manifFilePath, 'wb') as manif:
+        for fname in fileManif:
+            manif.write(fname)
+            manif.write('\n')
-- 
2.6.2



^ permalink raw reply related	[flat|nested] 7+ messages in thread