* [layerindex-web][PATCH 0/3] Bugfix, tests, doc update
@ 2018-07-19 13:17 Paul Eggleton
2018-07-19 13:17 ` [layerindex-web][PATCH 1/3] update: fix handling of moves outside of a layer Paul Eggleton
` (2 more replies)
0 siblings, 3 replies; 4+ messages in thread
From: Paul Eggleton @ 2018-07-19 13:17 UTC (permalink / raw
To: yocto
A fix for a bug in the update script, some regression tests (finally!)
and a small bit for the README about setting up dependencies using
virtualenv.
The following changes since commit 49981aebf6fab9338f2a570acbb9aa67312c8865:
Add site-wide notice support (2018-07-09 13:50:15 +0200)
are available in the Git repository at:
git://git.yoctoproject.org/layerindex-web paule/fixes4
http://git.yoctoproject.org/cgit.cgi//log/?h=paule/fixes4
Paul Eggleton (3):
update: fix handling of moves outside of a layer
Add minimal tests for update script
README: add an example virtualenv-based setup
.gitignore | 1 +
README | 37 ++++-
layerindex/tools/fixup_duplicates.py | 69 ++++++++++
layerindex/update_layer.py | 27 ++--
pytest.ini | 4 +
tests/test_update.py | 195 +++++++++++++++++++++++++++
6 files changed, 322 insertions(+), 11 deletions(-)
create mode 100644 layerindex/tools/fixup_duplicates.py
create mode 100644 pytest.ini
create mode 100644 tests/test_update.py
--
2.17.1
^ permalink raw reply [flat|nested] 4+ messages in thread
* [layerindex-web][PATCH 1/3] update: fix handling of moves outside of a layer
2018-07-19 13:17 [layerindex-web][PATCH 0/3] Bugfix, tests, doc update Paul Eggleton
@ 2018-07-19 13:17 ` Paul Eggleton
2018-07-19 13:17 ` [layerindex-web][PATCH 2/3] Add minimal tests for update script Paul Eggleton
2018-07-19 13:17 ` [layerindex-web][PATCH 3/3] README: add an example virtualenv-based setup Paul Eggleton
2 siblings, 0 replies; 4+ messages in thread
From: Paul Eggleton @ 2018-07-19 13:17 UTC (permalink / raw
To: yocto
If a file is moved (renamed) to a path outside of the layer, e.g.
another layer within a multi-layer repository, then we need to treat it
as a delete. Up until now we were updating the path and continuing, and
then the recipe was also picked up as an add in the other layer, leading
to duplicate recipe entries. I'd noticed these duplicates before but up
until now I'd thought that they were due to another bug we already
fixed, apparently not.
In order to remove these erroneous duplicate entries in existing
databases I have also added a layerindex/tools/fixup_duplicates.py
script. I've also made the -r/--reload option delete them as well.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
layerindex/tools/fixup_duplicates.py | 69 ++++++++++++++++++++++++++++
layerindex/update_layer.py | 27 +++++++----
2 files changed, 87 insertions(+), 9 deletions(-)
create mode 100644 layerindex/tools/fixup_duplicates.py
diff --git a/layerindex/tools/fixup_duplicates.py b/layerindex/tools/fixup_duplicates.py
new file mode 100644
index 00000000..5abe2c2d
--- /dev/null
+++ b/layerindex/tools/fixup_duplicates.py
@@ -0,0 +1,69 @@
+#!/usr/bin/env python3
+
+# Fix recipes that were moved out
+#
+# Copyright (C) 2017 Intel Corporation
+# Author: Paul Eggleton <paul.eggleton@linux.intel.com>
+#
+# Licensed under the MIT license, see COPYING.MIT for details
+
+
+import sys
+import os
+
+sys.path.insert(0, os.path.realpath(os.path.join(os.path.dirname(__file__), '..')))
+
+import optparse
+import utils
+import logging
+
+class DryRunRollbackException(Exception):
+ pass
+
+logger = utils.logger_create('LayerIndexFixup')
+
+
+
+def main():
+ parser = optparse.OptionParser(
+ usage = """
+ %prog [options""")
+
+ parser.add_option("-n", "--dry-run",
+ help = "Don't write any data back to the database",
+ action="store_true", dest="dryrun")
+ parser.add_option("-d", "--debug",
+ help = "Enable debug output",
+ action="store_const", const=logging.DEBUG, dest="loglevel", default=logging.INFO)
+ parser.add_option("-q", "--quiet",
+ help = "Hide all output except error messages",
+ action="store_const", const=logging.ERROR, dest="loglevel")
+
+ options, args = parser.parse_args(sys.argv)
+
+ utils.setup_django()
+ import settings
+ from layerindex.models import Recipe
+ from django.db import transaction
+
+ logger.setLevel(options.loglevel)
+
+ try:
+ with transaction.atomic():
+ #LayerBranch.objects.filter(layermaintainer__isnull=True).delete()
+ #LayerItem.objects.filter(layerbranch__isnull=True).filter(classic=False).delete()
+ #LayerItem.objects.filter(layerbranch__isnull=True).filter(classic=False).delete()
+ for recipe in Recipe.objects.filter(filepath__startswith='../'):
+ print('Deleting erroneous recipe %s %s' % (recipe.layerbranch, recipe))
+ recipe.delete()
+
+ if options.dryrun:
+ raise DryRunRollbackException()
+ except DryRunRollbackException:
+ pass
+
+ sys.exit(0)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/layerindex/update_layer.py b/layerindex/update_layer.py
index d34d8a59..2b39e8ef 100644
--- a/layerindex/update_layer.py
+++ b/layerindex/update_layer.py
@@ -502,6 +502,10 @@ def main():
if skip:
continue
if oldpath.startswith(subdir_start):
+ if not newpath.startswith(subdir_start):
+ logger.debug("Treating rename of %s to %s as a delete since new path is outside layer" % (oldpath, newpath))
+ other_deletes.append(diffitem)
+ continue
(oldtypename, oldfilepath, oldfilename) = recipeparse.detect_file_type(oldpath, subdir_start)
(newtypename, newfilepath, newfilename) = recipeparse.detect_file_type(newpath, subdir_start)
if oldtypename != newtypename:
@@ -684,16 +688,21 @@ def main():
# First, check which recipes still exist
layerrecipe_values = layerrecipes.values('id', 'filepath', 'filename', 'pn')
for v in layerrecipe_values:
- root = os.path.join(layerdir, v['filepath'])
- fullpath = os.path.join(root, v['filename'])
- preserve = True
- if os.path.exists(fullpath):
- for removedir in removedirs:
- if fullpath.startswith(removedir):
- preserve = False
- break
- else:
+ if v['filepath'].startswith('../'):
+ # FIXME: These recipes were present due to a bug (not handling renames
+ # to paths outside the layer) - this can be removed at some point in the future
preserve = False
+ else:
+ root = os.path.join(layerdir, v['filepath'])
+ fullpath = os.path.join(root, v['filename'])
+ if os.path.exists(fullpath):
+ preserve = True
+ for removedir in removedirs:
+ if fullpath.startswith(removedir):
+ preserve = False
+ break
+ else:
+ preserve = False
if preserve:
# Recipe still exists, update it
--
2.17.1
^ permalink raw reply related [flat|nested] 4+ messages in thread
* [layerindex-web][PATCH 2/3] Add minimal tests for update script
2018-07-19 13:17 [layerindex-web][PATCH 0/3] Bugfix, tests, doc update Paul Eggleton
2018-07-19 13:17 ` [layerindex-web][PATCH 1/3] update: fix handling of moves outside of a layer Paul Eggleton
@ 2018-07-19 13:17 ` Paul Eggleton
2018-07-19 13:17 ` [layerindex-web][PATCH 3/3] README: add an example virtualenv-based setup Paul Eggleton
2 siblings, 0 replies; 4+ messages in thread
From: Paul Eggleton @ 2018-07-19 13:17 UTC (permalink / raw
To: yocto
Up to this point we haven't had any regression tests for the layer
index, but the application (in particular the update script) has become
rather complicated, and we have had a few regressions, so here are some
tests. I've implemented them using pytest and pytest-django; I chose
pytest since we are starting from scratch here and I like the idea of
pytest's fixtures among other features. Annoyingly though because of our
use of separate scripts that need to perform database operations I had
to hack around some of the behaviour of pytest-django, which is clearly
not designed with this kind of structure in mind, but that's taken care
of now. Note that I've only considered backend testing for the moment,
there's not yet a strategy for testing the UI.
To run the tests you simply run "pytest" in the root of the repository.
You will need to have a working configuration set in settings.py (a
database needs to be set, but won't actually be used), and if you're
using MariaDB/MySQL then you'll need the READ COMMITTED transaction
isolation mode selected.
At the moment there are only a few basic tests for the update script
and a bunch of comments describing some we should add. The tests use
a newly added synthetic meta-layerindex-test layer on
git.yoctoproject.org so we can have something with known and fairly
static content. I expect we will extend these tests in the near future.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
.gitignore | 1 +
pytest.ini | 4 +
tests/test_update.py | 195 +++++++++++++++++++++++++++++++++++++++++++
3 files changed, 200 insertions(+)
create mode 100644 pytest.ini
create mode 100644 tests/test_update.py
diff --git a/.gitignore b/.gitignore
index 0010cdb6..edfc2ac1 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,4 +1,5 @@
*.pyc
*.db3
*.swp
+.pytest_cache
venv/
diff --git a/pytest.ini b/pytest.ini
new file mode 100644
index 00000000..c067b059
--- /dev/null
+++ b/pytest.ini
@@ -0,0 +1,4 @@
+[pytest]
+DJANGO_SETTINGS_MODULE = settings
+norecursedirs = .git layers
+
diff --git a/tests/test_update.py b/tests/test_update.py
new file mode 100644
index 00000000..e00775d4
--- /dev/null
+++ b/tests/test_update.py
@@ -0,0 +1,195 @@
+# layerindex-web - tests for update script
+#
+# Copyright (C) 2018 Intel Corporation
+#
+# Licensed under the MIT license, see COPYING.MIT for details
+
+# NOTE: requires pytest-django. Run using "pytest" from the root
+# of the repository (add -s to avoid suppressing the output of commands
+# when working on the tests)
+
+# NOTE: for these tests to work with MySQL / MariaDB and Django 1.11, you will need
+# to set the transaction isolation mode to READ COMMITTED (basically set
+# transaction-isolation = READ-COMMITTED in the [mysqld] section of /etc/my.cnf)
+
+# NOTE: we cannot practically save and restore the database between tests (since
+# we want these tests to work with any database backend) nor use transactions
+# (since we need to launch the update script which uses a separate database
+# connection), thus these tests cannot depend upon eachother - i.e. they must
+# not touch the test layer(s) in a manner that would affect any other test. In
+# practice that means separate recipes for each test.
+
+import sys
+import os
+import shutil
+import subprocess
+import pytest
+
+basepath = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
+
+def run_cmd(cmd, cwd=None):
+ if not cwd:
+ cwd = basepath
+ subprocess.check_call(cmd, stderr=subprocess.STDOUT, shell=True, cwd=cwd)
+
+@pytest.fixture(autouse=True)
+def enable_db_access_for_all_tests(db):
+ pass
+
+@pytest.fixture
+def db_access_without_rollback_and_truncate(request, django_db_setup, django_db_blocker):
+ django_db_blocker.unblock()
+ request.addfinalizer(django_db_blocker.restore)
+
+@pytest.fixture(scope="module")
+def backup_settings(tmpdir_factory):
+ stmpdir = tmpdir_factory.mktemp('settings')
+ settingsfile = os.path.join(basepath, 'settings.py')
+ backupfile = os.path.join(stmpdir, 'settings.bak')
+ shutil.copy(settingsfile, backupfile)
+ yield settingsfile
+ shutil.copy(backupfile, settingsfile)
+
+@pytest.fixture(scope="module")
+def hack_settings(backup_settings):
+ # It's horrific to have to do this, but we need to have additional
+ # scripts connect to the testing database and not whatever's in settings.py
+ # on disk right now, and this appears to be the only real way to do that
+ from django.conf import settings
+ with open(backup_settings, 'a') as f:
+ f.write('\nDATABASES = %s\n' % settings.DATABASES)
+
+@pytest.fixture(scope="module")
+def import_layer(hack_settings):
+ run_cmd("layerindex/tools/import_layer.py git://git.openembedded.org/openembedded-core -s meta openembedded-core")
+ run_cmd("layerindex/tools/import_layer.py git://git.yoctoproject.org/meta-layerindex-test -s meta-layerindex-test")
+ run_cmd("layerindex/update.py -l meta-layerindex-test")
+
+def test_example_recipe(import_layer):
+ from layerindex.models import Branch, LayerItem, LayerBranch
+ layer = LayerItem.objects.get(name='meta-layerindex-test')
+ layerbranch = LayerBranch.objects.get(layer__name='meta-layerindex-test')
+ found = False
+ for recipe in layerbranch.recipe_set.all():
+ if recipe.pn == 'example':
+ if found:
+ assert False, 'Duplicate example recipe in database'
+ assert recipe.pv == '0.1'
+ assert recipe.summary == 'Example recipe'
+ assert recipe.description == 'An example recipe used to test the OE layer index update script'
+ assert recipe.license == 'MIT'
+ assert recipe.filename == 'example_0.1.bb'
+ assert recipe.filepath == 'recipes-example/example'
+ # section is currently relying on what's set by bitbake.conf
+ assert recipe.section == 'base'
+ assert recipe.provides.split() == ['example']
+ assert recipe.blacklisted == ''
+ # homepage bugtracker bbclassextend inherits
+ # dependencies
+ found = True
+ if not found:
+ assert False, "Expected 'example' recipe not in database"
+
+@pytest.fixture()
+def repo(db_access_without_rollback_and_truncate):
+ from layerindex.models import LayerItem
+ from django.conf import settings
+ fetchdir = settings.LAYER_FETCH_DIR
+ layer = LayerItem.objects.get(name='meta-layerindex-test')
+ urldir = layer.get_fetch_dir()
+ repodir = os.path.join(fetchdir, urldir)
+ yield repodir
+
+def test_move_recipe_out(import_layer, repo, db_access_without_rollback_and_truncate):
+ from layerindex.models import LayerBranch, Recipe
+ layerbranch = LayerBranch.objects.get(layer__name='meta-layerindex-test')
+ recipe = layerbranch.recipe_set.filter(pn='moveme').first()
+ assert recipe, 'No moveme recipe found'
+ os.makedirs(os.path.join(repo, 'meta-layerindex-othertest/recipes-somethingelse/moveme'))
+ run_cmd('tree', cwd=repo)
+ run_cmd('git mv meta-layerindex-test/recipes-example/moveme/moveme_0.1.bb meta-layerindex-othertest/recipes-somethingelse/moveme/moveme_0.1.bb', cwd=repo)
+ run_cmd('git commit -m "Move recipe to a different layer"', cwd=repo)
+ run_cmd("layerindex/update.py -d -l meta-layerindex-test --nofetch --nocheckout")
+ # Recipe should have been deleted by update script
+ with pytest.raises(Recipe.DoesNotExist):
+ recipe.refresh_from_db()
+
+def test_delete_recipe(import_layer, repo, db_access_without_rollback_and_truncate):
+ from layerindex.models import LayerBranch, Recipe
+ layerbranch = LayerBranch.objects.get(layer__name='meta-layerindex-test')
+ recipe = layerbranch.recipe_set.filter(pn='deleteme').first()
+ assert recipe, 'No deleteme recipe found'
+ run_cmd('git rm meta-layerindex-test/recipes-example/deleteme/deleteme_0.1.bb', cwd=repo)
+ run_cmd('git commit -m "Delete recipe"', cwd=repo)
+ run_cmd("layerindex/update.py -d -l meta-layerindex-test --nofetch --nocheckout")
+ # Recipe should have been deleted by update script
+ with pytest.raises(Recipe.DoesNotExist):
+ recipe.refresh_from_db()
+
+def test_upgrade_recipe(import_layer, repo, db_access_without_rollback_and_truncate):
+ # FIXME this test is a little simplistic
+ from layerindex.models import LayerBranch, Recipe
+ layerbranch = LayerBranch.objects.get(layer__name='meta-layerindex-test')
+ recipe = Recipe.objects.filter(layerbranch=layerbranch, pn='upgrademe').first()
+ assert recipe, 'No upgrademe recipe found'
+ oldid = recipe.id
+ run_cmd('git mv meta-layerindex-test/recipes-example/upgrademe/upgrademe_0.1.bb meta-layerindex-test/recipes-example/upgrademe/upgrademe_0.2.bb', cwd=repo)
+ run_cmd('git commit -m "Upgrade recipe"', cwd=repo)
+ run_cmd("layerindex/update.py -d -l meta-layerindex-test --nofetch --nocheckout")
+ recipe = Recipe.objects.filter(layerbranch=layerbranch, pn='upgrademe').first()
+ assert recipe.id == oldid
+ assert recipe.pv == "0.2"
+ assert recipe.filename == 'upgrademe_0.2.bb'
+ assert recipe.filepath == 'recipes-example/upgrademe'
+
+def test_add_recipe(import_layer, repo, db_access_without_rollback_and_truncate):
+ from layerindex.models import LayerBranch, Recipe
+ layerbranch = LayerBranch.objects.get(layer__name='meta-layerindex-test')
+ recipe = Recipe.objects.filter(layerbranch=layerbranch, pn='addme').first()
+ assert not recipe, 'addme recipe found when it should not have been'
+ os.makedirs(os.path.join(repo, 'meta-layerindex-test/recipes-example/addme'))
+ with open(os.path.join(repo, 'meta-layerindex-test/recipes-example/addme/addme_0.5.bb'), 'w') as f:
+ f.write('SUMMARY = "Brand new recipe"\n')
+ f.write('LICENSE = "MIT"\n')
+ run_cmd('git add meta-layerindex-test/recipes-example/addme/addme_0.5.bb', cwd=repo)
+ run_cmd('git commit -m "Add recipe"', cwd=repo)
+ run_cmd("layerindex/update.py -d -l meta-layerindex-test --nofetch --nocheckout")
+ # Recipe should have been deleted by update script
+ recipe = Recipe.objects.filter(layerbranch=layerbranch, pn='addme').first()
+ assert recipe, 'addme recipe not found'
+ assert recipe.pv == "0.5"
+ assert recipe.summary == "Brand new recipe"
+ assert recipe.license == "MIT"
+
+
+
+# FIXME test recipe modify
+# FIXME test recipe upgrade with inc merge?
+# FIXME test patches
+# FIXME test sources
+# FIXME test inherits
+
+# FIXME test distro add
+# FIXME test distro rename
+# FIXME test distro modify
+# FIXME test distro delete
+
+# FIXME test machine add
+# FIXME test machine rename
+# FIXME test machine modify
+# FIXME test machine delete
+
+# FIXME test bbappend add
+# FIXME test bbappend rename
+# FIXME test bbappend delete
+
+# FIXME test bbclass add
+# FIXME test bbclass rename
+# FIXME test bbclass delete
+
+# FIXME test modify bbclass updates dependent recipes
+# FIXME test modify inc updates dependent recipes
+
+# FIXME 'adding' a group of layers 'out of order' e.g. layer1 - require layer 5 and 4.. layer 2 - require layer 3 and 4, layer 3 - require layer 5, layer 4 require layer 5, layer 5 (no requirements)
+
+# FIXME test REST API (different module!)
--
2.17.1
^ permalink raw reply related [flat|nested] 4+ messages in thread
* [layerindex-web][PATCH 3/3] README: add an example virtualenv-based setup
2018-07-19 13:17 [layerindex-web][PATCH 0/3] Bugfix, tests, doc update Paul Eggleton
2018-07-19 13:17 ` [layerindex-web][PATCH 1/3] update: fix handling of moves outside of a layer Paul Eggleton
2018-07-19 13:17 ` [layerindex-web][PATCH 2/3] Add minimal tests for update script Paul Eggleton
@ 2018-07-19 13:17 ` Paul Eggleton
2 siblings, 0 replies; 4+ messages in thread
From: Paul Eggleton @ 2018-07-19 13:17 UTC (permalink / raw
To: yocto
A while ago I was trying to help someone new get started with a
development setup for the layer index and it occurred to me that the
virtualenv-based environment I am using for development isn't actually
covered in the documentation, so I wrote something quick for them but
didn't do anything further with it until now. Here it is in more fleshed
out form for the README.
Also clarify the python requirements for the update script a little bit.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
README | 37 +++++++++++++++++++++++++++++++++++--
1 file changed, 35 insertions(+), 2 deletions(-)
diff --git a/README b/README
index 72dce71d..d1b1f9bd 100644
--- a/README
+++ b/README
@@ -29,8 +29,41 @@ In order to make use of this application you will need:
in settings.py and have access to the database used by the web
application):
* Python 2.7.6+ / Python 3.4+ to match with the version of BitBake
- for the OpenEmbedded branch being parsed
- * GitPython (python-git) version 2.0 or later
+ for the OpenEmbedded branch being parsed (for modern versions it's
+ Python 3.)
+ * Python dependencies as per requirements.txt (we still need Django
+ etc. here since we interact with the database through Django's ORM.)
+
+Example virtualenv-based setup for the above:
+
+Python's virtualenv provides an easy way to isolate the python dependencies
+of applications such as the layer index. Here's an example of setting up a
+virtualenv for the layer index that's particularly useful for development.
+(This assumes a Debian-based distribution, adjust accordingly for other
+distros).
+
+1. Install required host distro packages (some of these are required by
+ pip to build the dependencies; it's also assumed you want MariaDB as
+ the database backend):
+
+ sudo apt-get install virtualenv libmariadb-client-lgpl-dev build-essential python3-dev libjpeg-dev libz-dev libfreetype6-dev mariadb-server rabbitmq-server
+
+2. Work around path issues (you may not need this):
+
+ sudo ln -s /usr/bin/mariadb_config /usr/bin/mysql_config
+
+3. Create a Python 3 virtualenv (path can be anywhere you like):
+
+ virtualenv -p python3 /path/to/desired/venv
+
+4. Activate the virtualenv:
+
+ . /path/to/desired/venv/bin/activate
+
+5. Install requirements:
+
+ pip install -r requirements.txt
+
Setup instructions:
--
2.17.1
^ permalink raw reply related [flat|nested] 4+ messages in thread
end of thread, other threads:[~2018-07-19 13:18 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2018-07-19 13:17 [layerindex-web][PATCH 0/3] Bugfix, tests, doc update Paul Eggleton
2018-07-19 13:17 ` [layerindex-web][PATCH 1/3] update: fix handling of moves outside of a layer Paul Eggleton
2018-07-19 13:17 ` [layerindex-web][PATCH 2/3] Add minimal tests for update script Paul Eggleton
2018-07-19 13:17 ` [layerindex-web][PATCH 3/3] README: add an example virtualenv-based setup Paul Eggleton
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.