* [PATCH 0/2] Improve externalsrc task dependency tracking
@ 2016-01-26 14:13 Markus Lehtonen
2016-01-26 14:13 ` [PATCH 1/2] externalsrc.bbclas: remove nostamp from do_configure Markus Lehtonen
2016-01-26 14:13 ` [PATCH 2/2] devtool: create-workspace: define separate cache for task file checksums Markus Lehtonen
0 siblings, 2 replies; 3+ messages in thread
From: Markus Lehtonen @ 2016-01-26 14:13 UTC (permalink / raw
To: openembedded-core
This patchset improves task hashing of do_compile when externalsrc is enabled.
Previously, it was simply a 'nostamp' task that was always being (re-)run. This
patchset changes externalsrc to utilize the file-checksum dependency feature of
tasks to really track changes in the source tree, and thus, aims in preventing
unneeded (re-)compilations.
[YOCTO #8853]
The following changes since commit fc4209baa098caebf9c4cb75f9a6f2e85f43333c:
Revert "xz: Allow to work with ASSUME_PROVIDED xz-native" (2016-01-25 10:08:25 +0000)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib marquiz/devtool/fixes
Markus Lehtonen (2):
externalsrc.bbclas: remove nostamp from do_configure
devtool: create-workspace: define separate cache for task file
checksums
meta/classes/externalsrc.bbclass | 3 +--
scripts/devtool | 1 +
2 files changed, 2 insertions(+), 2 deletions(-)
--
2.1.4
^ permalink raw reply [flat|nested] 3+ messages in thread
* [PATCH 1/2] externalsrc.bbclas: remove nostamp from do_configure
2016-01-26 14:13 [PATCH 0/2] Improve externalsrc task dependency tracking Markus Lehtonen
@ 2016-01-26 14:13 ` Markus Lehtonen
2016-01-26 14:13 ` [PATCH 2/2] devtool: create-workspace: define separate cache for task file checksums Markus Lehtonen
1 sibling, 0 replies; 3+ messages in thread
From: Markus Lehtonen @ 2016-01-26 14:13 UTC (permalink / raw
To: openembedded-core
Be a bit more intelligent than mindlessly re-compiling every time.
Instead of using 'nostamp' flag for do_compile add the whole source tree
as 'file-checksums' flag. This way, do_compile is only re-run if
something in the source tree content changes. Hidden files and
directories in the source tree root are ignored by the glob currently
used. This has the advantage of automatically ignoring .git directory,
for example.
This does not work perfectly, though, as many packages are built under
${S} which effectively changes the source tree causing some unwanted
re-compilations. However, if do_compile of the recipe does not produce
new/different artefacts on every run (as commonly is and should be the
case) the re-compilation loop stops. Thus, you should usually see only
one re-compilation (if any) after which the source tree is "stabilized"
and no more re-compilations happen.
During the first bitbake run preparing of the task runqueue may take
much longer because all the files in the source tree are hashed.
Subsequent builds are not significantly slower because (most) file
hashes are found from the cache.
[YOCTO #8853]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
meta/classes/externalsrc.bbclass | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
index f7ed66d..fe4963d 100644
--- a/meta/classes/externalsrc.bbclass
+++ b/meta/classes/externalsrc.bbclass
@@ -83,8 +83,7 @@ python () {
d.prependVarFlag('do_compile', 'prefuncs', "externalsrc_compile_prefunc ")
- # Ensure compilation happens every time
- d.setVarFlag('do_compile', 'nostamp', '1')
+ d.setVarFlag('do_compile', 'file-checksums', externalsrc + '/*:True')
}
python externalsrc_compile_prefunc() {
--
2.1.4
^ permalink raw reply related [flat|nested] 3+ messages in thread
* [PATCH 2/2] devtool: create-workspace: define separate cache for task file checksums
2016-01-26 14:13 [PATCH 0/2] Improve externalsrc task dependency tracking Markus Lehtonen
2016-01-26 14:13 ` [PATCH 1/2] externalsrc.bbclas: remove nostamp from do_configure Markus Lehtonen
@ 2016-01-26 14:13 ` Markus Lehtonen
1 sibling, 0 replies; 3+ messages in thread
From: Markus Lehtonen @ 2016-01-26 14:13 UTC (permalink / raw
To: openembedded-core
Define BB_HASH_CHECKSUM_CACHE_FILE variable in the workspace layer
config so that a specific cache is used for storing the task file
dependency checksums. The file checksum cache can grow quite large after
hashing of all file in externalsrc trees was enabled. This patch
prevents polluting/growing the fetcher local file checksum cache. Also,
this new devtool-specific cache is not used after the workspace layer is
disabled.
[YOCTO #8853]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
scripts/devtool | 1 +
1 file changed, 1 insertion(+)
diff --git a/scripts/devtool b/scripts/devtool
index 2d57da0..e44b87e9 100755
--- a/scripts/devtool
+++ b/scripts/devtool
@@ -152,6 +152,7 @@ def _create_workspace(workspacedir, config, basepath):
f.write('BBFILE_PATTERN_workspacelayer = "^$' + '{LAYERDIR}/"\n')
f.write('BBFILE_PATTERN_IGNORE_EMPTY_workspacelayer = "1"\n')
f.write('BBFILE_PRIORITY_workspacelayer = "99"\n')
+ f.write('BB_HASH_CHECKSUM_CACHE_FILE = "filedep_checksum_cache.dat"\n')
# Add a README file
with open(os.path.join(workspacedir, 'README'), 'w') as f:
f.write('This layer was created by the OpenEmbedded devtool utility in order to\n')
--
2.1.4
^ permalink raw reply related [flat|nested] 3+ messages in thread
end of thread, other threads:[~2016-01-26 14:13 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-01-26 14:13 [PATCH 0/2] Improve externalsrc task dependency tracking Markus Lehtonen
2016-01-26 14:13 ` [PATCH 1/2] externalsrc.bbclas: remove nostamp from do_configure Markus Lehtonen
2016-01-26 14:13 ` [PATCH 2/2] devtool: create-workspace: define separate cache for task file checksums Markus Lehtonen
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.