* [Qemu-devel] [PATCH v5 0/5] migration: Dynamic cpu throttling for auto-converge
@ 2015-09-01 14:45 Jason J. Herne
2015-09-01 14:45 ` [Qemu-devel] [PATCH v5 1/5] cpu: Provide vcpu throttling interface Jason J. Herne
` (4 more replies)
0 siblings, 5 replies; 14+ messages in thread
From: Jason J. Herne @ 2015-09-01 14:45 UTC (permalink / raw
To: afaerber, amit.shah, dgilbert, borntraeger, quintela, qemu-devel,
pbonzini
Cc: Jason J. Herne
This patch set provides a new method for throttling a vcpu and makes use of said
method to dynamically increase cpu throttling during an autoconverge
migration until the migration completes.
This work is related to the following discussion:
https://lists.gnu.org/archive/html/qemu-devel/2015-03/msg00287.html
Previous version review is here:
https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg00734.html
Changelog
-----------
v5
- Add cpu->throttle_thread_scheduled flag to prevent callbacks from stacking up
- Correct error in formula used to reset the timer
- Add atomic operations to protect throttle_percentage
- Make use of cpu_throttle_get_percentage() instead of accessing data directly
- Various small formatting/style fixes
v4
- Switch to a static timer
- Use QEMU_CLOCK_VIRTUAL_RT instead of QEMU_CLOCK_REALTIME
- Get rid of throttle_timer_stop, use throttle_percentage = 0 instead
- Calculate throttle_ratio directly in cpu_throttle_thread
- Consolidate timer mod operations to single function
- Remove some unneeded cpu_throttle_active() checks
- Check for throttle_percentage == 0 in cpu_throttle_thread
- Change throttle_percentage to unsigned int
- Renamed cpu_throttle_timer_pop to cpu_throttle_timer_tick
v3
- Cpu throttling parameters moved from CPUState to global scope
- Throttling interface now uses a percentage instead of ratio
- Migration parameters added to allow tweaking of how aggressive throttling is
- Add throttle active check to the throttle stop routine.
- Remove asserts from throttle start/stop functions and instead force the input
to fall within the acceptable range
- Rename cpu_throttle_start to cpu_throttle_set to better describe its purpose
- Remove unneeded object casts
- Fixed monitor output formatting
- Comment cleanups
v2
- Add cpu throttle ratio output to hmp/qmp info/query migrate commands
v1
- Initial
Jason J. Herne (5):
cpu: Provide vcpu throttling interface
migration: Parameters for auto-converge cpu throttling
migration: Dynamic cpu throttling for auto-converge
qmp/hmp: Add throttle ratio to query-migrate and info migrate
migration: Disambiguate MAX_THROTTLE
arch_init.c | 88 ++++++++++++++++++---------------------------------
cpus.c | 78 +++++++++++++++++++++++++++++++++++++++++++++
hmp.c | 21 ++++++++++++
include/qom/cpu.h | 43 +++++++++++++++++++++++++
migration/migration.c | 57 +++++++++++++++++++++++++++++++--
qapi-schema.json | 40 ++++++++++++++++++++---
6 files changed, 263 insertions(+), 64 deletions(-)
--
1.9.1
^ permalink raw reply [flat|nested] 14+ messages in thread
* [Qemu-devel] [PATCH v5 1/5] cpu: Provide vcpu throttling interface
2015-09-01 14:45 [Qemu-devel] [PATCH v5 0/5] migration: Dynamic cpu throttling for auto-converge Jason J. Herne
@ 2015-09-01 14:45 ` Jason J. Herne
2015-09-01 21:23 ` Eric Blake
2015-09-01 14:46 ` [Qemu-devel] [PATCH v5 2/5] migration: Parameters for auto-converge cpu throttling Jason J. Herne
` (3 subsequent siblings)
4 siblings, 1 reply; 14+ messages in thread
From: Jason J. Herne @ 2015-09-01 14:45 UTC (permalink / raw
To: afaerber, amit.shah, dgilbert, borntraeger, quintela, qemu-devel,
pbonzini
Cc: Jason J. Herne
Provide a method to throttle guest cpu execution. CPUState is augmented with
timeout controls and throttle start/stop functions. To throttle the guest cpu
the caller simply has to call the throttle set function and provide a percentage
of throttle time.
Signed-off-by: Jason J. Herne <jjherne@linux.vnet.ibm.com>
Reviewed-by: Matthew Rosato <mjrosato@linux.vnet.ibm.com>
---
cpus.c | 78 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
include/qom/cpu.h | 43 ++++++++++++++++++++++++++++++
2 files changed, 121 insertions(+)
diff --git a/cpus.c b/cpus.c
index de6469f..f504e72 100644
--- a/cpus.c
+++ b/cpus.c
@@ -68,6 +68,14 @@ static CPUState *next_cpu;
int64_t max_delay;
int64_t max_advance;
+/* vcpu throttling controls */
+static QEMUTimer *throttle_timer;
+static unsigned int throttle_percentage;
+
+#define CPU_THROTTLE_PCT_MIN 1
+#define CPU_THROTTLE_PCT_MAX 99
+#define CPU_THROTTLE_TIMESLICE 10
+
bool cpu_is_stopped(CPUState *cpu)
{
return cpu->stopped || !runstate_is_running();
@@ -486,10 +494,80 @@ static const VMStateDescription vmstate_timers = {
}
};
+static void cpu_throttle_thread(void *opaque)
+{
+ CPUState *cpu = opaque;
+ double pct;
+ double throttle_ratio;
+ long sleeptime_ms;
+
+ if (!cpu_throttle_get_percentage()) {
+ return;
+ }
+
+ pct = (double)cpu_throttle_get_percentage()/100;
+ throttle_ratio = pct / (1 - pct);
+ sleeptime_ms = (long)(throttle_ratio * CPU_THROTTLE_TIMESLICE);
+
+ qemu_mutex_unlock_iothread();
+ atomic_set(&cpu->throttle_thread_scheduled, 0);
+ g_usleep(sleeptime_ms * 1000); /* Convert ms to us for usleep call */
+ qemu_mutex_lock_iothread();
+}
+
+static void cpu_throttle_timer_tick(void *opaque)
+{
+ CPUState *cpu;
+ double pct;
+
+ /* Stop the timer if needed */
+ if (!cpu_throttle_get_percentage()) {
+ return;
+ }
+ CPU_FOREACH(cpu) {
+ if (!atomic_xchg(&cpu->throttle_thread_scheduled, 1)) {
+ async_run_on_cpu(cpu, cpu_throttle_thread, cpu);
+ }
+ }
+
+ pct = (double)cpu_throttle_get_percentage()/100;
+ timer_mod(throttle_timer, qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL_RT) +
+ CPU_THROTTLE_TIMESLICE / (1-pct));
+}
+
+void cpu_throttle_set(int new_throttle_pct)
+{
+ /* Ensure throttle percentage is within valid range */
+ new_throttle_pct = MIN(new_throttle_pct, CPU_THROTTLE_PCT_MAX);
+ new_throttle_pct = MAX(new_throttle_pct, CPU_THROTTLE_PCT_MIN);
+
+ atomic_set(&throttle_percentage, new_throttle_pct);
+
+ timer_mod(throttle_timer, qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL_RT) +
+ CPU_THROTTLE_TIMESLICE);
+}
+
+void cpu_throttle_stop(void)
+{
+ atomic_set(&throttle_percentage, 0);
+}
+
+bool cpu_throttle_active(void)
+{
+ return (cpu_throttle_get_percentage() != 0);
+}
+
+int cpu_throttle_get_percentage(void)
+{
+ return atomic_read(&throttle_percentage);
+}
+
void cpu_ticks_init(void)
{
seqlock_init(&timers_state.vm_clock_seqlock, NULL);
vmstate_register(NULL, 0, &vmstate_timers, &timers_state);
+ throttle_timer = timer_new_ms(QEMU_CLOCK_VIRTUAL_RT,
+ cpu_throttle_timer_tick, NULL);
}
void configure_icount(QemuOpts *opts, Error **errp)
diff --git a/include/qom/cpu.h b/include/qom/cpu.h
index 39f0f19..fdca10c 100644
--- a/include/qom/cpu.h
+++ b/include/qom/cpu.h
@@ -310,6 +310,11 @@ struct CPUState {
uint32_t can_do_io;
int32_t exception_index; /* used by m68k TCG */
+ /* Used to keep track of an outstanding cpu throttle thread for migration
+ * autoconverge
+ */
+ bool throttle_thread_scheduled;
+
/* Note that this is accessed at the start of every TB via a negative
offset from AREG0. Leave this field at the end so as to make the
(absolute value) offset as small as possible. This reduces code
@@ -553,6 +558,44 @@ CPUState *qemu_get_cpu(int index);
*/
bool cpu_exists(int64_t id);
+/**
+ * cpu_throttle_set:
+ * @new_throttle_pct: Percent of sleep time to running time.
+ * Valid range is 1 to 99.
+ *
+ * Throttles all vcpus by forcing them to sleep for the given percentage of
+ * time. A throttle_percentage of 50 corresponds to a 50% duty cycle roughly.
+ * (example: 10ms sleep for every 10ms awake).
+ *
+ * cpu_throttle_set can be called as needed to adjust new_throttle_pct.
+ * Once the throttling starts, it will remain in effect until cpu_throttle_stop
+ * is called.
+ */
+void cpu_throttle_set(int new_throttle_pct);
+
+/**
+ * cpu_throttle_stop:
+ *
+ * Stops the vcpu throttling started by cpu_throttle_set.
+ */
+void cpu_throttle_stop(void);
+
+/**
+ * cpu_throttle_active:
+ *
+ * Returns %true if the vcpus are currently being throttled, %false otherwise.
+ */
+bool cpu_throttle_active(void);
+
+/**
+ * cpu_throttle_get_percentage:
+ *
+ * Returns the vcpu throttle percentage. See cpu_throttle_set for details.
+ *
+ * Returns The throttle percentage in range 1 to 99.
+ */
+int cpu_throttle_get_percentage(void);
+
#ifndef CONFIG_USER_ONLY
typedef void (*CPUInterruptHandler)(CPUState *, int);
--
1.9.1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [Qemu-devel] [PATCH v5 2/5] migration: Parameters for auto-converge cpu throttling
2015-09-01 14:45 [Qemu-devel] [PATCH v5 0/5] migration: Dynamic cpu throttling for auto-converge Jason J. Herne
2015-09-01 14:45 ` [Qemu-devel] [PATCH v5 1/5] cpu: Provide vcpu throttling interface Jason J. Herne
@ 2015-09-01 14:46 ` Jason J. Herne
2015-09-01 21:26 ` Eric Blake
2015-09-01 14:46 ` [Qemu-devel] [PATCH v5 3/5] migration: Dynamic cpu throttling for auto-converge Jason J. Herne
` (2 subsequent siblings)
4 siblings, 1 reply; 14+ messages in thread
From: Jason J. Herne @ 2015-09-01 14:46 UTC (permalink / raw
To: afaerber, amit.shah, dgilbert, borntraeger, quintela, qemu-devel,
pbonzini
Cc: Jason J. Herne
Add migration parameters to allow the user to adjust the parameters
that control cpu throttling when auto-converge is in effect. The added
parameters are as follows:
x-cpu-throttle-initial : Initial percantage of time guest cpus are throttled
when migration auto-converge is activated.
x-cpu-throttle-increment: throttle percantage increase each time
auto-converge detects that migration is not making progress.
Signed-off-by: Jason J. Herne <jjherne@linux.vnet.ibm.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
---
hmp.c | 16 ++++++++++++++++
migration/migration.c | 46 +++++++++++++++++++++++++++++++++++++++++++++-
qapi-schema.json | 33 ++++++++++++++++++++++++++++++---
3 files changed, 91 insertions(+), 4 deletions(-)
diff --git a/hmp.c b/hmp.c
index e17852d..eb65998 100644
--- a/hmp.c
+++ b/hmp.c
@@ -269,6 +269,12 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)
monitor_printf(mon, " %s: %" PRId64,
MigrationParameter_lookup[MIGRATION_PARAMETER_DECOMPRESS_THREADS],
params->decompress_threads);
+ monitor_printf(mon, " %s: %" PRId64,
+ MigrationParameter_lookup[MIGRATION_PARAMETER_X_CPU_THROTTLE_INITIAL],
+ params->x_cpu_throttle_initial);
+ monitor_printf(mon, " %s: %" PRId64,
+ MigrationParameter_lookup[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT],
+ params->x_cpu_throttle_increment);
monitor_printf(mon, "\n");
}
@@ -1216,6 +1222,8 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
bool has_compress_level = false;
bool has_compress_threads = false;
bool has_decompress_threads = false;
+ bool has_x_cpu_throttle_initial = false;
+ bool has_x_cpu_throttle_increment = false;
int i;
for (i = 0; i < MIGRATION_PARAMETER_MAX; i++) {
@@ -1230,10 +1238,18 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
case MIGRATION_PARAMETER_DECOMPRESS_THREADS:
has_decompress_threads = true;
break;
+ case MIGRATION_PARAMETER_X_CPU_THROTTLE_INITIAL:
+ has_x_cpu_throttle_initial = true;
+ break;
+ case MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT:
+ has_x_cpu_throttle_increment = true;
+ break;
}
qmp_migrate_set_parameters(has_compress_level, value,
has_compress_threads, value,
has_decompress_threads, value,
+ has_x_cpu_throttle_initial, value,
+ has_x_cpu_throttle_increment, value,
&err);
break;
}
diff --git a/migration/migration.c b/migration/migration.c
index 732d229..05790e9 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -40,6 +40,9 @@
#define DEFAULT_MIGRATE_DECOMPRESS_THREAD_COUNT 2
/*0: means nocompress, 1: best speed, ... 9: best compress ratio */
#define DEFAULT_MIGRATE_COMPRESS_LEVEL 1
+/* Define default autoconverge cpu throttle migration parameters */
+#define DEFAULT_MIGRATE_X_CPU_THROTTLE_INITIAL 20
+#define DEFAULT_MIGRATE_X_CPU_THROTTLE_INCREMENT 10
/* Migration XBZRLE default cache size */
#define DEFAULT_MIGRATE_CACHE_SIZE (64 * 1024 * 1024)
@@ -66,6 +69,10 @@ MigrationState *migrate_get_current(void)
DEFAULT_MIGRATE_COMPRESS_THREAD_COUNT,
.parameters[MIGRATION_PARAMETER_DECOMPRESS_THREADS] =
DEFAULT_MIGRATE_DECOMPRESS_THREAD_COUNT,
+ .parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INITIAL] =
+ DEFAULT_MIGRATE_X_CPU_THROTTLE_INITIAL,
+ .parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT] =
+ DEFAULT_MIGRATE_X_CPU_THROTTLE_INCREMENT,
};
return ¤t_migration;
@@ -199,6 +206,10 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)
s->parameters[MIGRATION_PARAMETER_COMPRESS_THREADS];
params->decompress_threads =
s->parameters[MIGRATION_PARAMETER_DECOMPRESS_THREADS];
+ params->x_cpu_throttle_initial =
+ s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INITIAL];
+ params->x_cpu_throttle_increment =
+ s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT];
return params;
}
@@ -321,7 +332,11 @@ void qmp_migrate_set_parameters(bool has_compress_level,
bool has_compress_threads,
int64_t compress_threads,
bool has_decompress_threads,
- int64_t decompress_threads, Error **errp)
+ int64_t decompress_threads,
+ bool has_x_cpu_throttle_initial,
+ int64_t x_cpu_throttle_initial,
+ bool has_x_cpu_throttle_increment,
+ int64_t x_cpu_throttle_increment, Error **errp)
{
MigrationState *s = migrate_get_current();
@@ -344,6 +359,18 @@ void qmp_migrate_set_parameters(bool has_compress_level,
"is invalid, it should be in the range of 1 to 255");
return;
}
+ if (has_x_cpu_throttle_initial &&
+ (x_cpu_throttle_initial < 1 || x_cpu_throttle_initial > 99)) {
+ error_set(errp, QERR_INVALID_PARAMETER_VALUE,
+ "x_cpu_throttle_initial",
+ "an integer in the range of 1 to 99");
+ }
+ if (has_x_cpu_throttle_increment &&
+ (x_cpu_throttle_increment < 1 || x_cpu_throttle_increment > 99)) {
+ error_set(errp, QERR_INVALID_PARAMETER_VALUE,
+ "x_cpu_throttle_increment",
+ "an integer in the range of 1 to 99");
+ }
if (has_compress_level) {
s->parameters[MIGRATION_PARAMETER_COMPRESS_LEVEL] = compress_level;
@@ -355,6 +382,15 @@ void qmp_migrate_set_parameters(bool has_compress_level,
s->parameters[MIGRATION_PARAMETER_DECOMPRESS_THREADS] =
decompress_threads;
}
+ if (has_x_cpu_throttle_initial) {
+ s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INITIAL] =
+ x_cpu_throttle_initial;
+ }
+
+ if (has_x_cpu_throttle_increment) {
+ s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT] =
+ x_cpu_throttle_increment;
+ }
}
/* shared migration helpers */
@@ -470,6 +506,10 @@ static MigrationState *migrate_init(const MigrationParams *params)
s->parameters[MIGRATION_PARAMETER_COMPRESS_THREADS];
int decompress_thread_count =
s->parameters[MIGRATION_PARAMETER_DECOMPRESS_THREADS];
+ int x_cpu_throttle_initial =
+ s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INITIAL];
+ int x_cpu_throttle_increment =
+ s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT];
memcpy(enabled_capabilities, s->enabled_capabilities,
sizeof(enabled_capabilities));
@@ -485,6 +525,10 @@ static MigrationState *migrate_init(const MigrationParams *params)
compress_thread_count;
s->parameters[MIGRATION_PARAMETER_DECOMPRESS_THREADS] =
decompress_thread_count;
+ s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INITIAL] =
+ x_cpu_throttle_initial;
+ s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT] =
+ x_cpu_throttle_increment;
s->bandwidth_limit = bandwidth_limit;
s->state = MIGRATION_STATUS_SETUP;
trace_migrate_set_state(MIGRATION_STATUS_SETUP);
diff --git a/qapi-schema.json b/qapi-schema.json
index f97ffa1..52b7e07 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -587,10 +587,18 @@
# compression, so set the decompress-threads to the number about 1/4
# of compress-threads is adequate.
#
+# @x-cpu-throttle-initial: Initial percentage of time guest cpus are throttled
+# when migration auto-converge is activated.
+# (Since 2.4)
+#
+# @x-cpu-throttle-increment: throttle percentage increase each time
+# auto-converge detects that migration is not making
+# progress. (Since 2.4)
# Since: 2.4
##
{ 'enum': 'MigrationParameter',
- 'data': ['compress-level', 'compress-threads', 'decompress-threads'] }
+ 'data': ['compress-level', 'compress-threads', 'decompress-threads',
+ 'x-cpu-throttle-initial', 'x-cpu-throttle-increment'] }
#
# @migrate-set-parameters
@@ -603,12 +611,22 @@
#
# @decompress-threads: decompression thread count
#
+# @x-cpu-throttle-initial: Initial percentage of time guest cpus are throttled
+# when migration auto-converge is activated.
+# (Since 2.4)
+#
+# @x-cpu-throttle-increment: throttle percentage increase each time
+# auto-converge detects that migration is not making
+# progress. (Since 2.4)
+#
# Since: 2.4
##
{ 'command': 'migrate-set-parameters',
'data': { '*compress-level': 'int',
'*compress-threads': 'int',
- '*decompress-threads': 'int'} }
+ '*decompress-threads': 'int',
+ '*x-cpu-throttle-initial': 'int',
+ '*x-cpu-throttle-increment': 'int'} }
#
# @MigrationParameters
@@ -618,13 +636,22 @@
# @compress-threads: compression thread count
#
# @decompress-threads: decompression thread count
+# @x-cpu-throttle-initial: Initial percentage of time guest cpus are throttled
+# when migration auto-converge is activated.
+# (Since 2.4)
+#
+# @x-cpu-throttle-increment: throttle percentage increase each time
+# auto-converge detects that migration is not making
+# progress. (Since 2.4)
#
# Since: 2.4
##
{ 'struct': 'MigrationParameters',
'data': { 'compress-level': 'int',
'compress-threads': 'int',
- 'decompress-threads': 'int'} }
+ 'decompress-threads': 'int',
+ 'x-cpu-throttle-initial': 'int',
+ 'x-cpu-throttle-increment': 'int'} }
##
# @query-migrate-parameters
#
--
1.9.1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [Qemu-devel] [PATCH v5 3/5] migration: Dynamic cpu throttling for auto-converge
2015-09-01 14:45 [Qemu-devel] [PATCH v5 0/5] migration: Dynamic cpu throttling for auto-converge Jason J. Herne
2015-09-01 14:45 ` [Qemu-devel] [PATCH v5 1/5] cpu: Provide vcpu throttling interface Jason J. Herne
2015-09-01 14:46 ` [Qemu-devel] [PATCH v5 2/5] migration: Parameters for auto-converge cpu throttling Jason J. Herne
@ 2015-09-01 14:46 ` Jason J. Herne
2015-09-01 21:30 ` Eric Blake
2015-09-01 14:46 ` [Qemu-devel] [PATCH v5 4/5] qmp/hmp: Add throttle ratio to query-migrate and info migrate Jason J. Herne
2015-09-01 14:46 ` [Qemu-devel] [PATCH v5 5/5] migration: Disambiguate MAX_THROTTLE Jason J. Herne
4 siblings, 1 reply; 14+ messages in thread
From: Jason J. Herne @ 2015-09-01 14:46 UTC (permalink / raw
To: afaerber, amit.shah, dgilbert, borntraeger, quintela, qemu-devel,
pbonzini
Cc: Jason J. Herne
Remove traditional auto-converge static 30ms throttling code and replace it
with a dynamic throttling algorithm.
Additionally, be more aggressive when deciding when to start throttling.
Previously we waited until four unproductive memory passes. Now we begin
throttling after only two unproductive memory passes. Four seemed quite
arbitrary and only waiting for two passes allows us to complete the migration
faster.
Signed-off-by: Jason J. Herne <jjherne@linux.vnet.ibm.com>
Reviewed-by: Matthew Rosato <mjrosato@linux.vnet.ibm.com>
---
arch_init.c | 88 ++++++++++++++++++---------------------------------
migration/migration.c | 4 +++
2 files changed, 34 insertions(+), 58 deletions(-)
diff --git a/arch_init.c b/arch_init.c
index 23d3feb..35f8eb0 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -111,9 +111,7 @@ int graphic_depth = 32;
#endif
const uint32_t arch_type = QEMU_ARCH;
-static bool mig_throttle_on;
static int dirty_rate_high_cnt;
-static void check_guest_throttling(void);
static uint64_t bitmap_sync_count;
@@ -487,6 +485,29 @@ static size_t save_page_header(QEMUFile *f, RAMBlock *block, ram_addr_t offset)
return size;
}
+/* Reduce amount of guest cpu execution to hopefully slow down memory writes.
+ * If guest dirty memory rate is reduced below the rate at which we can
+ * transfer pages to the destination then we should be able to complete
+ * migration. Some workloads dirty memory way too fast and will not effectively
+ * converge, even with auto-converge.
+ */
+static void mig_throttle_guest_down(void)
+{
+ MigrationState *s = migrate_get_current();
+ uint64_t pct_initial =
+ s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INITIAL];
+ uint64_t pct_icrement =
+ s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT];
+
+ /* We have not started throttling yet. Let's start it. */
+ if (!cpu_throttle_active()) {
+ cpu_throttle_set(pct_initial);
+ } else {
+ /* Throttling already on, just increase the rate */
+ cpu_throttle_set(cpu_throttle_get_percentage() + pct_icrement);
+ }
+}
+
/* Update the xbzrle cache to reflect a page that's been sent as all 0.
* The important thing is that a stale (not-yet-0'd) page be replaced
* by the new data.
@@ -714,21 +735,21 @@ static void migration_bitmap_sync(void)
/* The following detection logic can be refined later. For now:
Check to see if the dirtied bytes is 50% more than the approx.
amount of bytes that just got transferred since the last time we
- were in this routine. If that happens >N times (for now N==4)
- we turn on the throttle down logic */
+ were in this routine. If that happens twice, start or increase
+ throttling */
bytes_xfer_now = ram_bytes_transferred();
+
if (s->dirty_pages_rate &&
(num_dirty_pages_period * TARGET_PAGE_SIZE >
(bytes_xfer_now - bytes_xfer_prev)/2) &&
- (dirty_rate_high_cnt++ > 4)) {
+ (dirty_rate_high_cnt++ >= 2)) {
trace_migration_throttle();
- mig_throttle_on = true;
dirty_rate_high_cnt = 0;
+ mig_throttle_guest_down();
}
bytes_xfer_prev = bytes_xfer_now;
- } else {
- mig_throttle_on = false;
}
+
if (migrate_use_xbzrle()) {
if (iterations_prev != acct_info.iterations) {
acct_info.xbzrle_cache_miss_rate =
@@ -1197,7 +1218,6 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
RAMBlock *block;
int64_t ram_bitmap_pages; /* Size of bitmap in pages, including gaps */
- mig_throttle_on = false;
dirty_rate_high_cnt = 0;
bitmap_sync_count = 0;
migration_bitmap_sync_init();
@@ -1301,7 +1321,7 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
}
pages_sent += pages;
acct_info.iterations++;
- check_guest_throttling();
+
/* we want to check in the 1st loop, just in case it was the 1st time
and we had to sync the dirty bitmap.
qemu_get_clock_ns() is a bit expensive, so we only check each some
@@ -1913,51 +1933,3 @@ TargetInfo *qmp_query_target(Error **errp)
return info;
}
-/* Stub function that's gets run on the vcpu when its brought out of the
- VM to run inside qemu via async_run_on_cpu()*/
-static void mig_sleep_cpu(void *opq)
-{
- qemu_mutex_unlock_iothread();
- g_usleep(30*1000);
- qemu_mutex_lock_iothread();
-}
-
-/* To reduce the dirty rate explicitly disallow the VCPUs from spending
- much time in the VM. The migration thread will try to catchup.
- Workload will experience a performance drop.
-*/
-static void mig_throttle_guest_down(void)
-{
- CPUState *cpu;
-
- qemu_mutex_lock_iothread();
- CPU_FOREACH(cpu) {
- async_run_on_cpu(cpu, mig_sleep_cpu, NULL);
- }
- qemu_mutex_unlock_iothread();
-}
-
-static void check_guest_throttling(void)
-{
- static int64_t t0;
- int64_t t1;
-
- if (!mig_throttle_on) {
- return;
- }
-
- if (!t0) {
- t0 = qemu_clock_get_ns(QEMU_CLOCK_REALTIME);
- return;
- }
-
- t1 = qemu_clock_get_ns(QEMU_CLOCK_REALTIME);
-
- /* If it has been more than 40 ms since the last time the guest
- * was throttled then do it again.
- */
- if (40 < (t1-t0)/1000000) {
- mig_throttle_guest_down();
- t0 = t1;
- }
-}
diff --git a/migration/migration.c b/migration/migration.c
index 05790e9..7708c54 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -25,6 +25,7 @@
#include "qemu/thread.h"
#include "qmp-commands.h"
#include "trace.h"
+#include "qom/cpu.h"
#define MAX_THROTTLE (32 << 20) /* Migration speed throttling */
@@ -858,6 +859,9 @@ static void *migration_thread(void *opaque)
}
}
+ /* If we enabled cpu throttling for auto-converge, turn it off. */
+ cpu_throttle_stop();
+
qemu_mutex_lock_iothread();
if (s->state == MIGRATION_STATUS_COMPLETED) {
int64_t end_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
--
1.9.1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [Qemu-devel] [PATCH v5 4/5] qmp/hmp: Add throttle ratio to query-migrate and info migrate
2015-09-01 14:45 [Qemu-devel] [PATCH v5 0/5] migration: Dynamic cpu throttling for auto-converge Jason J. Herne
` (2 preceding siblings ...)
2015-09-01 14:46 ` [Qemu-devel] [PATCH v5 3/5] migration: Dynamic cpu throttling for auto-converge Jason J. Herne
@ 2015-09-01 14:46 ` Jason J. Herne
2015-09-01 21:31 ` Eric Blake
2015-09-01 14:46 ` [Qemu-devel] [PATCH v5 5/5] migration: Disambiguate MAX_THROTTLE Jason J. Herne
4 siblings, 1 reply; 14+ messages in thread
From: Jason J. Herne @ 2015-09-01 14:46 UTC (permalink / raw
To: afaerber, amit.shah, dgilbert, borntraeger, quintela, qemu-devel,
pbonzini
Cc: Jason J. Herne
Report throttle percentage in info migrate and query-migrate responses when
cpu throttling is active.
Signed-off-by: Jason J. Herne <jjherne@linux.vnet.ibm.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
---
hmp.c | 5 +++++
migration/migration.c | 5 +++++
qapi-schema.json | 7 ++++++-
3 files changed, 16 insertions(+), 1 deletion(-)
diff --git a/hmp.c b/hmp.c
index eb65998..36bc76e 100644
--- a/hmp.c
+++ b/hmp.c
@@ -229,6 +229,11 @@ void hmp_info_migrate(Monitor *mon, const QDict *qdict)
info->xbzrle_cache->overflow);
}
+ if (info->has_x_cpu_throttle_percentage) {
+ monitor_printf(mon, "cpu throttle percentage: %" PRIu64 "\n",
+ info->x_cpu_throttle_percentage);
+ }
+
qapi_free_MigrationInfo(info);
qapi_free_MigrationCapabilityStatusList(caps);
}
diff --git a/migration/migration.c b/migration/migration.c
index 7708c54..b29450a 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -274,6 +274,11 @@ MigrationInfo *qmp_query_migrate(Error **errp)
info->disk->total = blk_mig_bytes_total();
}
+ if (cpu_throttle_active()) {
+ info->has_x_cpu_throttle_percentage = true;
+ info->x_cpu_throttle_percentage = cpu_throttle_get_percentage();
+ }
+
get_xbzrle_cache_stats(info);
break;
case MIGRATION_STATUS_COMPLETED:
diff --git a/qapi-schema.json b/qapi-schema.json
index 52b7e07..c2bd8ce 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -474,6 +474,10 @@
# may be expensive, but do not actually occur during the iterative
# migration rounds themselves. (since 1.6)
#
+# @x-cpu-throttle-percentage: #optional percentage of time guest cpus are being
+# throttled during auto-converge. This is only present when auto-converge
+# has started throttling guest cpus. (Since 2.4)
+#
# Since: 0.14.0
##
{ 'struct': 'MigrationInfo',
@@ -483,7 +487,8 @@
'*total-time': 'int',
'*expected-downtime': 'int',
'*downtime': 'int',
- '*setup-time': 'int'} }
+ '*setup-time': 'int',
+ '*x-cpu-throttle-percentage': 'int'} }
##
# @query-migrate
--
1.9.1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [Qemu-devel] [PATCH v5 5/5] migration: Disambiguate MAX_THROTTLE
2015-09-01 14:45 [Qemu-devel] [PATCH v5 0/5] migration: Dynamic cpu throttling for auto-converge Jason J. Herne
` (3 preceding siblings ...)
2015-09-01 14:46 ` [Qemu-devel] [PATCH v5 4/5] qmp/hmp: Add throttle ratio to query-migrate and info migrate Jason J. Herne
@ 2015-09-01 14:46 ` Jason J. Herne
2015-09-01 21:32 ` Eric Blake
4 siblings, 1 reply; 14+ messages in thread
From: Jason J. Herne @ 2015-09-01 14:46 UTC (permalink / raw
To: afaerber, amit.shah, dgilbert, borntraeger, quintela, qemu-devel,
pbonzini
Cc: Jason J. Herne
Migration has a define for MAX_THROTTLE. Update comment to clarify that this is
used for throttling transfer speed. Hopefully this will prevent it from being
confused with a guest cpu throttling entity.
Signed-off-by: Jason J. Herne <jjherne@linux.vnet.ibm.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
---
migration/migration.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/migration/migration.c b/migration/migration.c
index b29450a..b9faeb0 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -27,7 +27,7 @@
#include "trace.h"
#include "qom/cpu.h"
-#define MAX_THROTTLE (32 << 20) /* Migration speed throttling */
+#define MAX_THROTTLE (32 << 20) /* Migration transfer speed throttling */
/* Amount of time to allocate to each "chunk" of bandwidth-throttled
* data. */
--
1.9.1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [Qemu-devel] [PATCH v5 1/5] cpu: Provide vcpu throttling interface
2015-09-01 14:45 ` [Qemu-devel] [PATCH v5 1/5] cpu: Provide vcpu throttling interface Jason J. Herne
@ 2015-09-01 21:23 ` Eric Blake
2015-09-04 15:17 ` Jason J. Herne
0 siblings, 1 reply; 14+ messages in thread
From: Eric Blake @ 2015-09-01 21:23 UTC (permalink / raw
To: Jason J. Herne, afaerber, amit.shah, dgilbert, borntraeger,
quintela, qemu-devel, pbonzini
[-- Attachment #1: Type: text/plain, Size: 1900 bytes --]
On 09/01/2015 08:45 AM, Jason J. Herne wrote:
> Provide a method to throttle guest cpu execution. CPUState is augmented with
> timeout controls and throttle start/stop functions. To throttle the guest cpu
> the caller simply has to call the throttle set function and provide a percentage
> of throttle time.
>
> Signed-off-by: Jason J. Herne <jjherne@linux.vnet.ibm.com>
> Reviewed-by: Matthew Rosato <mjrosato@linux.vnet.ibm.com>
> ---
> cpus.c | 78 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
> include/qom/cpu.h | 43 ++++++++++++++++++++++++++++++
> 2 files changed, 121 insertions(+)
>
> +/**
> + * cpu_throttle_set:
> + * @new_throttle_pct: Percent of sleep time to running time.
I don't think you mean the ratio of sleep time to running time, as much
as sleep time to overall time.
That is, 10% should mean "sleep 1ms and run 9ms out of every 10ms
total", and not "sleep 1ms after every 10ms run, for 90.9% duty cycle".
> + * Valid range is 1 to 99.
> + *
> + * Throttles all vcpus by forcing them to sleep for the given percentage of
> + * time. A throttle_percentage of 50 corresponds to a 50% duty cycle roughly.
> + * (example: 10ms sleep for every 10ms awake).
50% can be a bit ambiguous (not obvious whether a higher percentage
means more sleep or more time awake); it might be better to pick a
different number, such as:
A throttle_percentage of 25 corresponds to a 75% duty cycle (example:
10ms sleep for every 30ms awake).
if the percentage really is [1 - duty cycle] (which I suspect), or:
A throttle_percentage of 25 corresponds to a 80% duty cycle (example:
10ms sleep for every 40ms awake).
(if I'm wrong, and you really meant percentage of sleep time to running
time).
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Qemu-devel] [PATCH v5 2/5] migration: Parameters for auto-converge cpu throttling
2015-09-01 14:46 ` [Qemu-devel] [PATCH v5 2/5] migration: Parameters for auto-converge cpu throttling Jason J. Herne
@ 2015-09-01 21:26 ` Eric Blake
2015-09-04 15:09 ` Jason J. Herne
0 siblings, 1 reply; 14+ messages in thread
From: Eric Blake @ 2015-09-01 21:26 UTC (permalink / raw
To: Jason J. Herne, afaerber, amit.shah, dgilbert, borntraeger,
quintela, qemu-devel, pbonzini
[-- Attachment #1: Type: text/plain, Size: 3476 bytes --]
On 09/01/2015 08:46 AM, Jason J. Herne wrote:
> Add migration parameters to allow the user to adjust the parameters
> that control cpu throttling when auto-converge is in effect. The added
> parameters are as follows:
>
> x-cpu-throttle-initial : Initial percantage of time guest cpus are throttled
> when migration auto-converge is activated.
>
> x-cpu-throttle-increment: throttle percantage increase each time
> auto-converge detects that migration is not making progress.
Marked experimental; do we think it will be promoted to stable before
2.5 is released?
What happens if I set throttle-initial to 50 and throttle-increment to
50? The previous patch required throttle percentage to be between 1 and
99 inclusive; does that mean I cap out at 99 instead of an invalid
throttle of 100?
>
> Signed-off-by: Jason J. Herne <jjherne@linux.vnet.ibm.com>
> Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> ---
> hmp.c | 16 ++++++++++++++++
> migration/migration.c | 46 +++++++++++++++++++++++++++++++++++++++++++++-
> qapi-schema.json | 33 ++++++++++++++++++++++++++++++---
> 3 files changed, 91 insertions(+), 4 deletions(-)
>
> +++ b/qapi-schema.json
> @@ -587,10 +587,18 @@
> # compression, so set the decompress-threads to the number about 1/4
> # of compress-threads is adequate.
> #
> +# @x-cpu-throttle-initial: Initial percentage of time guest cpus are throttled
> +# when migration auto-converge is activated.
> +# (Since 2.4)
> +#
> +# @x-cpu-throttle-increment: throttle percentage increase each time
> +# auto-converge detects that migration is not making
> +# progress. (Since 2.4)
We've missed 2.4; these lines need to say (Since 2.5).
> # Since: 2.4
> ##
> { 'enum': 'MigrationParameter',
> - 'data': ['compress-level', 'compress-threads', 'decompress-threads'] }
> + 'data': ['compress-level', 'compress-threads', 'decompress-threads',
> + 'x-cpu-throttle-initial', 'x-cpu-throttle-increment'] }
>
> #
> # @migrate-set-parameters
> @@ -603,12 +611,22 @@
> #
> # @decompress-threads: decompression thread count
> #
> +# @x-cpu-throttle-initial: Initial percentage of time guest cpus are throttled
> +# when migration auto-converge is activated.
> +# (Since 2.4)
Would be nice to document the default (20).
> +#
> +# @x-cpu-throttle-increment: throttle percentage increase each time
> +# auto-converge detects that migration is not making
> +# progress. (Since 2.4)
And again (10). Also, same comment about needing 2.5.
> @@ -618,13 +636,22 @@
> # @compress-threads: compression thread count
> #
> # @decompress-threads: decompression thread count
> +# @x-cpu-throttle-initial: Initial percentage of time guest cpus are throttled
> +# when migration auto-converge is activated.
> +# (Since 2.4)
> +#
> +# @x-cpu-throttle-increment: throttle percentage increase each time
> +# auto-converge detects that migration is not making
> +# progress. (Since 2.4)
2.5.
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Qemu-devel] [PATCH v5 3/5] migration: Dynamic cpu throttling for auto-converge
2015-09-01 14:46 ` [Qemu-devel] [PATCH v5 3/5] migration: Dynamic cpu throttling for auto-converge Jason J. Herne
@ 2015-09-01 21:30 ` Eric Blake
2015-09-04 14:54 ` Jason J. Herne
0 siblings, 1 reply; 14+ messages in thread
From: Eric Blake @ 2015-09-01 21:30 UTC (permalink / raw
To: Jason J. Herne, afaerber, amit.shah, dgilbert, borntraeger,
quintela, qemu-devel, pbonzini
[-- Attachment #1: Type: text/plain, Size: 1608 bytes --]
On 09/01/2015 08:46 AM, Jason J. Herne wrote:
> Remove traditional auto-converge static 30ms throttling code and replace it
> with a dynamic throttling algorithm.
>
> Additionally, be more aggressive when deciding when to start throttling.
> Previously we waited until four unproductive memory passes. Now we begin
> throttling after only two unproductive memory passes. Four seemed quite
> arbitrary and only waiting for two passes allows us to complete the migration
> faster.
>
> Signed-off-by: Jason J. Herne <jjherne@linux.vnet.ibm.com>
> Reviewed-by: Matthew Rosato <mjrosato@linux.vnet.ibm.com>
> ---
> arch_init.c | 88 ++++++++++++++++++---------------------------------
> migration/migration.c | 4 +++
> 2 files changed, 34 insertions(+), 58 deletions(-)
> +static void mig_throttle_guest_down(void)
> +{
> + MigrationState *s = migrate_get_current();
> + uint64_t pct_initial =
> + s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INITIAL];
> + uint64_t pct_icrement =
> + s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT];
> +
> + /* We have not started throttling yet. Let's start it. */
> + if (!cpu_throttle_active()) {
> + cpu_throttle_set(pct_initial);
> + } else {
> + /* Throttling already on, just increase the rate */
> + cpu_throttle_set(cpu_throttle_get_percentage() + pct_icrement);
What happens if the addition results in a percentage larger than 99?
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Qemu-devel] [PATCH v5 4/5] qmp/hmp: Add throttle ratio to query-migrate and info migrate
2015-09-01 14:46 ` [Qemu-devel] [PATCH v5 4/5] qmp/hmp: Add throttle ratio to query-migrate and info migrate Jason J. Herne
@ 2015-09-01 21:31 ` Eric Blake
0 siblings, 0 replies; 14+ messages in thread
From: Eric Blake @ 2015-09-01 21:31 UTC (permalink / raw
To: Jason J. Herne, afaerber, amit.shah, dgilbert, borntraeger,
quintela, qemu-devel, pbonzini
[-- Attachment #1: Type: text/plain, Size: 991 bytes --]
On 09/01/2015 08:46 AM, Jason J. Herne wrote:
> Report throttle percentage in info migrate and query-migrate responses when
> cpu throttling is active.
>
> Signed-off-by: Jason J. Herne <jjherne@linux.vnet.ibm.com>
> Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> ---
> hmp.c | 5 +++++
> migration/migration.c | 5 +++++
> qapi-schema.json | 7 ++++++-
> 3 files changed, 16 insertions(+), 1 deletion(-)
>
> +++ b/qapi-schema.json
> @@ -474,6 +474,10 @@
> # may be expensive, but do not actually occur during the iterative
> # migration rounds themselves. (since 1.6)
> #
> +# @x-cpu-throttle-percentage: #optional percentage of time guest cpus are being
> +# throttled during auto-converge. This is only present when auto-converge
> +# has started throttling guest cpus. (Since 2.4)
2.5
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Qemu-devel] [PATCH v5 5/5] migration: Disambiguate MAX_THROTTLE
2015-09-01 14:46 ` [Qemu-devel] [PATCH v5 5/5] migration: Disambiguate MAX_THROTTLE Jason J. Herne
@ 2015-09-01 21:32 ` Eric Blake
0 siblings, 0 replies; 14+ messages in thread
From: Eric Blake @ 2015-09-01 21:32 UTC (permalink / raw
To: Jason J. Herne, afaerber, amit.shah, dgilbert, borntraeger,
quintela, qemu-devel, pbonzini
[-- Attachment #1: Type: text/plain, Size: 634 bytes --]
On 09/01/2015 08:46 AM, Jason J. Herne wrote:
> Migration has a define for MAX_THROTTLE. Update comment to clarify that this is
> used for throttling transfer speed. Hopefully this will prevent it from being
> confused with a guest cpu throttling entity.
>
> Signed-off-by: Jason J. Herne <jjherne@linux.vnet.ibm.com>
> Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> ---
> migration/migration.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
Reviewed-by: Eric Blake <eblake@redhat.com>
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Qemu-devel] [PATCH v5 3/5] migration: Dynamic cpu throttling for auto-converge
2015-09-01 21:30 ` Eric Blake
@ 2015-09-04 14:54 ` Jason J. Herne
0 siblings, 0 replies; 14+ messages in thread
From: Jason J. Herne @ 2015-09-04 14:54 UTC (permalink / raw
To: Eric Blake, afaerber, amit.shah, dgilbert, borntraeger, quintela,
qemu-devel, pbonzini
On 09/01/2015 05:30 PM, Eric Blake wrote:
> On 09/01/2015 08:46 AM, Jason J. Herne wrote:
>> Remove traditional auto-converge static 30ms throttling code and replace it
>> with a dynamic throttling algorithm.
>>
>> Additionally, be more aggressive when deciding when to start throttling.
>> Previously we waited until four unproductive memory passes. Now we begin
>> throttling after only two unproductive memory passes. Four seemed quite
>> arbitrary and only waiting for two passes allows us to complete the migration
>> faster.
>>
>> Signed-off-by: Jason J. Herne <jjherne@linux.vnet.ibm.com>
>> Reviewed-by: Matthew Rosato <mjrosato@linux.vnet.ibm.com>
>> ---
>> arch_init.c | 88 ++++++++++++++++++---------------------------------
>> migration/migration.c | 4 +++
>> 2 files changed, 34 insertions(+), 58 deletions(-)
>
>> +static void mig_throttle_guest_down(void)
>> +{
>> + MigrationState *s = migrate_get_current();
>> + uint64_t pct_initial =
>> + s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INITIAL];
>> + uint64_t pct_icrement =
>> + s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT];
>> +
>> + /* We have not started throttling yet. Let's start it. */
>> + if (!cpu_throttle_active()) {
>> + cpu_throttle_set(pct_initial);
>> + } else {
>> + /* Throttling already on, just increase the rate */
>> + cpu_throttle_set(cpu_throttle_get_percentage() + pct_icrement);
>
> What happens if the addition results in a percentage larger than 99?
>
cpu_throttle_set takes responsibility for upper/lower bounds capping.
+void cpu_throttle_set(int new_throttle_pct)
+{
+ /* Ensure throttle percentage is within valid range */
+ new_throttle_pct = MIN(new_throttle_pct, CPU_THROTTLE_PCT_MAX);
+ new_throttle_pct = MAX(new_throttle_pct, CPU_THROTTLE_PCT_MIN);
+
+ atomic_set(&throttle_percentage, new_throttle_pct);
+
+ timer_mod(throttle_timer, qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL_RT) +
+ CPU_THROTTLE_TIMESLICE);
+}
+
--
-- Jason J. Herne (jjherne@linux.vnet.ibm.com)
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Qemu-devel] [PATCH v5 2/5] migration: Parameters for auto-converge cpu throttling
2015-09-01 21:26 ` Eric Blake
@ 2015-09-04 15:09 ` Jason J. Herne
0 siblings, 0 replies; 14+ messages in thread
From: Jason J. Herne @ 2015-09-04 15:09 UTC (permalink / raw
To: Eric Blake, afaerber, amit.shah, dgilbert, borntraeger, quintela,
qemu-devel, pbonzini
On 09/01/2015 05:26 PM, Eric Blake wrote:
> On 09/01/2015 08:46 AM, Jason J. Herne wrote:
>> Add migration parameters to allow the user to adjust the parameters
>> that control cpu throttling when auto-converge is in effect. The added
>> parameters are as follows:
>>
>> x-cpu-throttle-initial : Initial percantage of time guest cpus are throttled
>> when migration auto-converge is activated.
>>
>> x-cpu-throttle-increment: throttle percantage increase each time
>> auto-converge detects that migration is not making progress.
>
> Marked experimental; do we think it will be promoted to stable before
> 2.5 is released?
>
Not a clue :) I have no idea how things move from x- to sans x-.
> What happens if I set throttle-initial to 50 and throttle-increment to
> 50? The previous patch required throttle percentage to be between 1 and
> 99 inclusive; does that mean I cap out at 99 instead of an invalid
> throttle of 100?
>
The call to cpu_throttle_set will take care of the capping. Details
provided in previous reply :)
...
> We've missed 2.4; these lines need to say (Since 2.5).
...
>> # Since: 2.4
...
> Would be nice to document the default (20).
...
> And again (10). Also, same comment about needing 2.5.
>
I will fix all of these.
--
-- Jason J. Herne (jjherne@linux.vnet.ibm.com)
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Qemu-devel] [PATCH v5 1/5] cpu: Provide vcpu throttling interface
2015-09-01 21:23 ` Eric Blake
@ 2015-09-04 15:17 ` Jason J. Herne
0 siblings, 0 replies; 14+ messages in thread
From: Jason J. Herne @ 2015-09-04 15:17 UTC (permalink / raw
To: Eric Blake, afaerber, amit.shah, dgilbert, borntraeger, quintela,
qemu-devel, pbonzini
On 09/01/2015 05:23 PM, Eric Blake wrote:
> On 09/01/2015 08:45 AM, Jason J. Herne wrote:
>> Provide a method to throttle guest cpu execution. CPUState is augmented with
>> timeout controls and throttle start/stop functions. To throttle the guest cpu
>> the caller simply has to call the throttle set function and provide a percentage
>> of throttle time.
>>
>> Signed-off-by: Jason J. Herne <jjherne@linux.vnet.ibm.com>
>> Reviewed-by: Matthew Rosato <mjrosato@linux.vnet.ibm.com>
>> ---
>> cpus.c | 78 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> include/qom/cpu.h | 43 ++++++++++++++++++++++++++++++
>> 2 files changed, 121 insertions(+)
>>
>
>> +/**
>> + * cpu_throttle_set:
>> + * @new_throttle_pct: Percent of sleep time to running time.
>
> I don't think you mean the ratio of sleep time to running time, as much
> as sleep time to overall time.
>
> That is, 10% should mean "sleep 1ms and run 9ms out of every 10ms
> total", and not "sleep 1ms after every 10ms run, for 90.9% duty cycle".
>
>> + * Valid range is 1 to 99.
>> + *
>> + * Throttles all vcpus by forcing them to sleep for the given percentage of
>> + * time. A throttle_percentage of 50 corresponds to a 50% duty cycle roughly.
>> + * (example: 10ms sleep for every 10ms awake).
>
> 50% can be a bit ambiguous (not obvious whether a higher percentage
> means more sleep or more time awake); it might be better to pick a
> different number, such as:
>
> A throttle_percentage of 25 corresponds to a 75% duty cycle (example:
> 10ms sleep for every 30ms awake).
>
> if the percentage really is [1 - duty cycle] (which I suspect), or:
>
> A throttle_percentage of 25 corresponds to a 80% duty cycle (example:
> 10ms sleep for every 40ms awake).
>
> (if I'm wrong, and you really meant percentage of sleep time to running
> time).
>
You are 100% correct here. This comment is a relic of an older version.
Nice find :)
I'll update the docs and incorporate your suggestions.
--
-- Jason J. Herne (jjherne@linux.vnet.ibm.com)
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2015-09-04 15:18 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-09-01 14:45 [Qemu-devel] [PATCH v5 0/5] migration: Dynamic cpu throttling for auto-converge Jason J. Herne
2015-09-01 14:45 ` [Qemu-devel] [PATCH v5 1/5] cpu: Provide vcpu throttling interface Jason J. Herne
2015-09-01 21:23 ` Eric Blake
2015-09-04 15:17 ` Jason J. Herne
2015-09-01 14:46 ` [Qemu-devel] [PATCH v5 2/5] migration: Parameters for auto-converge cpu throttling Jason J. Herne
2015-09-01 21:26 ` Eric Blake
2015-09-04 15:09 ` Jason J. Herne
2015-09-01 14:46 ` [Qemu-devel] [PATCH v5 3/5] migration: Dynamic cpu throttling for auto-converge Jason J. Herne
2015-09-01 21:30 ` Eric Blake
2015-09-04 14:54 ` Jason J. Herne
2015-09-01 14:46 ` [Qemu-devel] [PATCH v5 4/5] qmp/hmp: Add throttle ratio to query-migrate and info migrate Jason J. Herne
2015-09-01 21:31 ` Eric Blake
2015-09-01 14:46 ` [Qemu-devel] [PATCH v5 5/5] migration: Disambiguate MAX_THROTTLE Jason J. Herne
2015-09-01 21:32 ` Eric Blake
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.