* [PATCH 1/4] cve-update-db-native: add progress handler
@ 2020-09-09 17:35 Chris Laplante
2020-09-09 17:35 ` [PATCH 2/4] cve-check/cve-update-db-native: fix under multiconfig Chris Laplante
` (2 more replies)
0 siblings, 3 replies; 7+ messages in thread
From: Chris Laplante @ 2020-09-09 17:35 UTC (permalink / raw
To: openembedded-core; +Cc: Chris Laplante
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
---
.../recipes-core/meta/cve-update-db-native.bb | 90 ++++++++++---------
1 file changed, 47 insertions(+), 43 deletions(-)
diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb
index 32d6dbdffc..2221825bf8 100644
--- a/meta/recipes-core/meta/cve-update-db-native.bb
+++ b/meta/recipes-core/meta/cve-update-db-native.bb
@@ -29,6 +29,7 @@ python do_populate_cve_db() {
Update NVD database with json data feed
"""
import bb.utils
+ import bb.progress
import sqlite3, urllib, urllib.parse, shutil, gzip
from datetime import date
@@ -60,54 +61,57 @@ python do_populate_cve_db() {
initialize_db(c)
- for year in range(YEAR_START, date.today().year + 1):
- year_url = BASE_URL + str(year)
- meta_url = year_url + ".meta"
- json_url = year_url + ".json.gz"
+ with bb.progress.ProgressHandler(d) as ph:
+ total_years = date.today().year + 1 - YEAR_START
+ for i, year in enumerate(range(YEAR_START, date.today().year + 1)):
+ ph.update((float(i + 1) / total_years) * 100)
+ year_url = BASE_URL + str(year)
+ meta_url = year_url + ".meta"
+ json_url = year_url + ".json.gz"
- # Retrieve meta last modified date
- try:
- response = urllib.request.urlopen(meta_url)
- except urllib.error.URLError as e:
- cve_f.write('Warning: CVE db update error, Unable to fetch CVE data.\n\n')
- bb.warn("Failed to fetch CVE data (%s)" % e.reason)
- return
-
- if response:
- for l in response.read().decode("utf-8").splitlines():
- key, value = l.split(":", 1)
- if key == "lastModifiedDate":
- last_modified = value
- break
- else:
- bb.warn("Cannot parse CVE metadata, update failed")
- return
-
- # Compare with current db last modified date
- c.execute("select DATE from META where YEAR = ?", (year,))
- meta = c.fetchone()
- if not meta or meta[0] != last_modified:
- # Clear products table entries corresponding to current year
- c.execute("delete from PRODUCTS where ID like ?", ('CVE-%d%%' % year,))
-
- # Update db with current year json file
+ # Retrieve meta last modified date
try:
- response = urllib.request.urlopen(json_url)
- if response:
- update_db(c, gzip.decompress(response.read()).decode('utf-8'))
- c.execute("insert or replace into META values (?, ?)", [year, last_modified])
+ response = urllib.request.urlopen(meta_url)
except urllib.error.URLError as e:
- cve_f.write('Warning: CVE db update error, CVE data is outdated.\n\n')
- bb.warn("Cannot parse CVE data (%s), update failed" % e.reason)
+ cve_f.write('Warning: CVE db update error, Unable to fetch CVE data.\n\n')
+ bb.warn("Failed to fetch CVE data (%s)" % e.reason)
return
- # Update success, set the date to cve_check file.
- if year == date.today().year:
- cve_f.write('CVE database update : %s\n\n' % date.today())
-
- cve_f.close()
- conn.commit()
- conn.close()
+ if response:
+ for l in response.read().decode("utf-8").splitlines():
+ key, value = l.split(":", 1)
+ if key == "lastModifiedDate":
+ last_modified = value
+ break
+ else:
+ bb.warn("Cannot parse CVE metadata, update failed")
+ return
+
+ # Compare with current db last modified date
+ c.execute("select DATE from META where YEAR = ?", (year,))
+ meta = c.fetchone()
+ if not meta or meta[0] != last_modified:
+ # Clear products table entries corresponding to current year
+ c.execute("delete from PRODUCTS where ID like ?", ('CVE-%d%%' % year,))
+
+ # Update db with current year json file
+ try:
+ response = urllib.request.urlopen(json_url)
+ if response:
+ update_db(c, gzip.decompress(response.read()).decode('utf-8'))
+ c.execute("insert or replace into META values (?, ?)", [year, last_modified])
+ except urllib.error.URLError as e:
+ cve_f.write('Warning: CVE db update error, CVE data is outdated.\n\n')
+ bb.warn("Cannot parse CVE data (%s), update failed" % e.reason)
+ return
+
+ # Update success, set the date to cve_check file.
+ if year == date.today().year:
+ cve_f.write('CVE database update : %s\n\n' % date.today())
+
+ cve_f.close()
+ conn.commit()
+ conn.close()
}
def initialize_db(c):
--
2.17.1
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH 2/4] cve-check/cve-update-db-native: fix under multiconfig
2020-09-09 17:35 [PATCH 1/4] cve-update-db-native: add progress handler Chris Laplante
@ 2020-09-09 17:35 ` Chris Laplante
2020-09-09 20:19 ` [OE-core] " Ross Burton
2020-09-09 17:35 ` [PATCH 3/4] cve-update-db-native: use context manager for cve_f Chris Laplante
2020-09-09 17:35 ` [PATCH 4/4] cve-check: avoid FileNotFoundError if no do_cve_check task has run Chris Laplante
2 siblings, 1 reply; 7+ messages in thread
From: Chris Laplante @ 2020-09-09 17:35 UTC (permalink / raw
To: openembedded-core; +Cc: Chris Laplante
Previously CVE_CHECK_DB_FILE / CVE_CHECK_DB_DIR was the same across
multiconfigs which led to a race condition wherein multiple
cve-update-db-native:do_populate_cve_db tasks could attempt to write to
the same sqlite database. This led to the following task failure:
Error executing a python function in exec_python_func() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:do_populate_cve_db(d)
0003:
File: '/mnt/data/agent/work/74f119cccb44f133/yocto/sources/poky/meta/recipes-core/meta/cve-update-db-native.bb', lineno: 103, function: do_populate_cve_db
0099: if year == date.today().year:
0100: cve_f.write('CVE database update : %s\n\n' % date.today())
0101:
0102: cve_f.close()
*** 0103: conn.commit()
0104: conn.close()
0105:}
0106:
0107:def initialize_db(c):
Exception: sqlite3.OperationalError: disk I/O error
The fix is to add a multiconfig-specific subdirectory to act as a layer of separation.
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
---
meta/classes/cve-check.bbclass | 2 +-
meta/recipes-core/meta/cve-update-db-native.bb | 3 +--
2 files changed, 2 insertions(+), 3 deletions(-)
diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
index 0889e7544a..485d147ef8 100644
--- a/meta/classes/cve-check.bbclass
+++ b/meta/classes/cve-check.bbclass
@@ -25,7 +25,7 @@
CVE_PRODUCT ??= "${BPN}"
CVE_VERSION ??= "${PV}"
-CVE_CHECK_DB_DIR ?= "${DL_DIR}/CVE_CHECK"
+CVE_CHECK_DB_DIR ?= "${DL_DIR}/${BB_CURRENT_MC}/CVE_CHECK"
CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/nvdcve_1.1.db"
CVE_CHECK_LOG ?= "${T}/cve.log"
diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb
index 2221825bf8..57368caf73 100644
--- a/meta/recipes-core/meta/cve-update-db-native.bb
+++ b/meta/recipes-core/meta/cve-update-db-native.bb
@@ -52,8 +52,7 @@ python do_populate_cve_db() {
cve_f = open(os.path.join(d.getVar("TMPDIR"), 'cve_check'), 'a')
- if not os.path.isdir(db_dir):
- os.mkdir(db_dir)
+ bb.utils.mkdirhier(db_dir)
# Connect to database
conn = sqlite3.connect(db_file)
--
2.17.1
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH 3/4] cve-update-db-native: use context manager for cve_f
2020-09-09 17:35 [PATCH 1/4] cve-update-db-native: add progress handler Chris Laplante
2020-09-09 17:35 ` [PATCH 2/4] cve-check/cve-update-db-native: fix under multiconfig Chris Laplante
@ 2020-09-09 17:35 ` Chris Laplante
2020-09-09 17:35 ` [PATCH 4/4] cve-check: avoid FileNotFoundError if no do_cve_check task has run Chris Laplante
2 siblings, 0 replies; 7+ messages in thread
From: Chris Laplante @ 2020-09-09 17:35 UTC (permalink / raw
To: openembedded-core; +Cc: Chris Laplante
---
meta/recipes-core/meta/cve-update-db-native.bb | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb
index 57368caf73..f8f13af97c 100644
--- a/meta/recipes-core/meta/cve-update-db-native.bb
+++ b/meta/recipes-core/meta/cve-update-db-native.bb
@@ -50,8 +50,6 @@ python do_populate_cve_db() {
except OSError:
pass
- cve_f = open(os.path.join(d.getVar("TMPDIR"), 'cve_check'), 'a')
-
bb.utils.mkdirhier(db_dir)
# Connect to database
@@ -60,7 +58,7 @@ python do_populate_cve_db() {
initialize_db(c)
- with bb.progress.ProgressHandler(d) as ph:
+ with bb.progress.ProgressHandler(d) as ph, open(os.path.join(d.getVar("TMPDIR"), 'cve_check'), 'a') as cve_f:
total_years = date.today().year + 1 - YEAR_START
for i, year in enumerate(range(YEAR_START, date.today().year + 1)):
ph.update((float(i + 1) / total_years) * 100)
@@ -108,7 +106,6 @@ python do_populate_cve_db() {
if year == date.today().year:
cve_f.write('CVE database update : %s\n\n' % date.today())
- cve_f.close()
conn.commit()
conn.close()
}
--
2.17.1
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH 4/4] cve-check: avoid FileNotFoundError if no do_cve_check task has run
2020-09-09 17:35 [PATCH 1/4] cve-update-db-native: add progress handler Chris Laplante
2020-09-09 17:35 ` [PATCH 2/4] cve-check/cve-update-db-native: fix under multiconfig Chris Laplante
2020-09-09 17:35 ` [PATCH 3/4] cve-update-db-native: use context manager for cve_f Chris Laplante
@ 2020-09-09 17:35 ` Chris Laplante
2 siblings, 0 replies; 7+ messages in thread
From: Chris Laplante @ 2020-09-09 17:35 UTC (permalink / raw
To: openembedded-core; +Cc: Chris Laplante
For example, if you just run 'bitbake cve-update-db-native' in a clean
build system, |cve_tmp_file| won't exist yet.
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
---
meta/classes/cve-check.bbclass | 13 +++++++------
1 file changed, 7 insertions(+), 6 deletions(-)
diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
index 485d147ef8..9f272cd60c 100644
--- a/meta/classes/cve-check.bbclass
+++ b/meta/classes/cve-check.bbclass
@@ -62,14 +62,15 @@ python cve_save_summary_handler () {
timestamp = datetime.datetime.now().strftime('%Y%m%d%H%M%S')
cve_summary_file = os.path.join(cvelogpath, "%s-%s.txt" % (cve_summary_name, timestamp))
- shutil.copyfile(cve_tmp_file, cve_summary_file)
+ if os.path.exists(cve_tmp_file):
+ shutil.copyfile(cve_tmp_file, cve_summary_file)
- if cve_summary_file and os.path.exists(cve_summary_file):
- cvefile_link = os.path.join(cvelogpath, cve_summary_name)
+ if cve_summary_file and os.path.exists(cve_summary_file):
+ cvefile_link = os.path.join(cvelogpath, cve_summary_name)
- if os.path.exists(os.path.realpath(cvefile_link)):
- os.remove(cvefile_link)
- os.symlink(os.path.basename(cve_summary_file), cvefile_link)
+ if os.path.exists(os.path.realpath(cvefile_link)):
+ os.remove(cvefile_link)
+ os.symlink(os.path.basename(cve_summary_file), cvefile_link)
}
addhandler cve_save_summary_handler
--
2.17.1
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [OE-core] [PATCH 2/4] cve-check/cve-update-db-native: fix under multiconfig
2020-09-09 17:35 ` [PATCH 2/4] cve-check/cve-update-db-native: fix under multiconfig Chris Laplante
@ 2020-09-09 20:19 ` Ross Burton
2020-09-09 20:31 ` Chris Laplante
0 siblings, 1 reply; 7+ messages in thread
From: Ross Burton @ 2020-09-09 20:19 UTC (permalink / raw
To: chris.laplante; +Cc: OE-core
On Wed, 9 Sep 2020 at 18:35, Chris Laplante via lists.openembedded.org
<chris.laplante=agilent.com@lists.openembedded.org> wrote:
>
> Previously CVE_CHECK_DB_FILE / CVE_CHECK_DB_DIR was the same across
> multiconfigs which led to a race condition wherein multiple
> cve-update-db-native:do_populate_cve_db tasks could attempt to write to
> the same sqlite database. This led to the following task failure:
But the databases will be identical so that's basically a waste of
time. How about adding a lock file so the second task waits for the
first to complete, and then does nothing as the database is up to
date?
Ross
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [OE-core] [PATCH 2/4] cve-check/cve-update-db-native: fix under multiconfig
2020-09-09 20:19 ` [OE-core] " Ross Burton
@ 2020-09-09 20:31 ` Chris Laplante
2020-09-09 20:46 ` Chris Laplante
0 siblings, 1 reply; 7+ messages in thread
From: Chris Laplante @ 2020-09-09 20:31 UTC (permalink / raw
To: Ross Burton; +Cc: OE-core
> On Wed, 9 Sep 2020 at 18:35, Chris Laplante via lists.openembedded.org
> <chris.laplante=agilent.com@lists.openembedded.org> wrote:
> >
> > Previously CVE_CHECK_DB_FILE / CVE_CHECK_DB_DIR was the same
> across
> > multiconfigs which led to a race condition wherein multiple
> > cve-update-db-native:do_populate_cve_db tasks could attempt to write
> > to the same sqlite database. This led to the following task failure:
>
> But the databases will be identical so that's basically a waste of time. How
> about adding a lock file so the second task waits for the first to complete,
> and then does nothing as the database is up to date?
That's a good point. I also considered just making it a bb.event.BuildStarted event handler, like how uninative works.
Chris
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [OE-core] [PATCH 2/4] cve-check/cve-update-db-native: fix under multiconfig
2020-09-09 20:31 ` Chris Laplante
@ 2020-09-09 20:46 ` Chris Laplante
0 siblings, 0 replies; 7+ messages in thread
From: Chris Laplante @ 2020-09-09 20:46 UTC (permalink / raw
To: Ross Burton; +Cc: OE-core
> > But the databases will be identical so that's basically a waste of
> > time. How about adding a lock file so the second task waits for the
> > first to complete, and then does nothing as the database is up to date?
>
> That's a good point. I also considered just making it a bb.event.BuildStarted
> event handler, like how uninative works.
Should have mentioned, I like your approach better and am working on a v2 patch series.
Thanks,
Chris
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2020-09-09 20:46 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-09-09 17:35 [PATCH 1/4] cve-update-db-native: add progress handler Chris Laplante
2020-09-09 17:35 ` [PATCH 2/4] cve-check/cve-update-db-native: fix under multiconfig Chris Laplante
2020-09-09 20:19 ` [OE-core] " Ross Burton
2020-09-09 20:31 ` Chris Laplante
2020-09-09 20:46 ` Chris Laplante
2020-09-09 17:35 ` [PATCH 3/4] cve-update-db-native: use context manager for cve_f Chris Laplante
2020-09-09 17:35 ` [PATCH 4/4] cve-check: avoid FileNotFoundError if no do_cve_check task has run Chris Laplante
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.