维护者：Steven Adriaensen。
以下列表考虑了与动态算法配置相关的论文。此列表绝非详尽无遗。如果您在此列表中找不到某篇论文，请告知我。
请注意，动态配置在许多不同的社区（使用许多不同的名称）中都有研究，并且每个社区都发展了略有不同的侧重点或评估标准。我们维护此文献列表的标准如下：
- 所提出的工作是否在目标算法运行时*即时*（即，在目标算法运行时）更改（超）参数？
- 这是否以自动化方式完成（例如，通过学习到的更新策略）？
- 它是否包含元学习组件（即，配置策略是否可以转移到它尚未“学习”的问题上）？
2023
41.
Sabbioni, Luca; Corda, Francesco; Restelli, Marcello
Stepsize Learning for Policy Gradient Methods in Contextual Markov Decision Processes 未发表
2023.
@unpublished{sabbioni-arxiv23a,
title = {Stepsize Learning for Policy Gradient Methods in Contextual Markov Decision Processes},
author = {Luca Sabbioni and Francesco Corda and Marcello Restelli},
url = {https://arxiv.org/abs/2306.07741},
year = {2023},
date = {2023-06-13},
abstract = {Policy-based algorithms are among the most widely adopted techniques in model-free RL, thanks to their strong theoretical groundings and good properties in continuous action spaces. Unfortunately, these methods require precise and problem-specific hyperparameter tuning to achieve good performance, and tend to struggle when asked to accomplish a series of heterogeneous tasks. In particular, the selection of the step size has a crucial impact on their ability to learn a highly performing policy, affecting the speed and the stability of the training process, and often being the main culprit for poor results. In this paper, we tackle these issues with a Meta Reinforcement Learning approach, by introducing a new formulation, known as meta-MDP, that can be used to solve any hyperparameter selection problem in RL with contextual processes. After providing a theoretical Lipschitz bound to the difference of performance in different tasks, we adopt the proposed framework to train a batch RL algorithm to dynamically recommend the most adequate step size for different policies and tasks. In conclusion, we present an experimental campaign to show the advantages of selecting an adaptive learning rate in heterogeneous environments.},
keywords = {},
pubstate = {published},
tppubtype = {unpublished}
}
基于策略的算法因其坚实的理论基础和在连续动作空间中的良好性能而成为无模型强化学习中最广泛采用的技术之一。然而，这些方法需要精确且针对特定问题的超参数调优才能获得良好性能，并且在被要求完成一系列异构任务时往往会遇到困难。特别是，步长（step size）的选择对其学习高性能策略的能力具有至关重要的影响，影响着训练过程的速度和稳定性，并且通常是导致结果不佳的主要原因。在本文中，我们通过引入一种称为 meta-MDP 的新方法，采用元强化学习（Meta Reinforcement Learning）方法来解决这些问题，该方法可用于解决具有上下文过程的强化学习中的任何超参数选择问题。在提供性能差异在不同任务上的理论 Lipschitz 界限后，我们采用所提出的框架来训练一个批处理强化学习算法，以动态地推荐最适合不同策略和任务的步长。最后，我们进行了一项实验研究，以展示在异构环境中选择自适应学习率的优势。
40.
Chen, Deyao; Buzdalov, Maxim; Doerr, Carola; Dang, Nguyen
Using Automated Algorithm Configuration for Parameter Control 会议
Proceedings of the 17th ACM/SIGEVO Conference on Foundations of Genetic Algorithms., 2023。
@conference{chen2023using,
title = {Using Automated Algorithm Configuration for Parameter Control},
author = {Deyao Chen and Maxim Buzdalov and Carola Doerr and Nguyen Dang},
url = {https://arxiv.org/abs/2302.12334},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
booktitle = {Proceedings of the 17th ACM/SIGEVO Conference on Foundations of Genetic Algorithms.},
journal = {arXiv preprint arXiv:2302.12334},
pages = {38-49},
abstract = {Dynamic Algorithm Configuration (DAC) tackles the question of how to automatically learn policies to control parameters of algorithms in a data-driven fashion. This question has received considerable attention from the evolutionary community in recent years. Having a good benchmark collection to gain structural understanding on the effectiveness and limitations of different solution methods for DAC is therefore strongly desirable. Following recent work on proposing DAC benchmarks with well-understood theoretical properties and ground truth information, in this work, we suggest as a new DAC benchmark the controlling of the key parameter λ in the (1+(λ,λ)) Genetic Algorithm for solving OneMax problems. We conduct a study on how to solve the DAC problem via the use of (static) automated algorithm configuration on the benchmark, and propose techniques to significantly improve the performance of the approach. Our approach is able to consistently outperform the default parameter control policy of the benchmark derived from previous theoretical work on sufficiently large problem sizes. We also present new findings on the landscape of the parameter-control search policies and propose methods to compute stronger baselines for the benchmark via numerical approximations of the true optimal policies.},
keywords = {},
pubstate = {published},
tppubtype = {conference}
}
动态算法配置 (DAC) 解决了如何以数据驱动的方式自动学习控制算法参数的策略这一问题。近年来，进化计算领域对此问题给予了相当大的关注。因此，拥有一个良好的基准集合来获得对不同解决方案方法的有效性和局限性的结构化理解是迫切需要的。继近期提出具有良好理论性质和真实信息（ground truth）的 DAC 基准工作之后，在本工作中，我们建议将 (1+(λ,λ)) 遗传算法用于解决 OneMax 问题的关键参数 λ 的控制作为一个新的 DAC 基准。我们研究了如何通过使用（静态）自动化算法配置来解决基准的 DAC 问题，并提出了显着提高该方法性能的技术。我们的方法在足够大的问题规模上能够持续优于先前理论工作推导出的基准默认参数控制策略。我们还提出了关于参数控制搜索策略景观的新发现，并提出了通过数值近似真实最优策略来计算该基准更强基线的方法。
2022
39.
Adriaensen, Steven; Biedenkapp, André; Shala, Gresa; Awad, Noor; Eimer, Theresa; Lindauer, Marius; Hutter, Frank
Automated Dynamic Algorithm Configuration 期刊文章
In: Journal of Artificial Intelligence Research (JAIR), vol. 75, pp. 1633-1699, 2022。
@article{adriaens-arxiv22a,
title = {Automated Dynamic Algorithm Configuration},
author = {Steven Adriaensen and André Biedenkapp and Gresa Shala and Noor Awad and Theresa Eimer and Marius Lindauer and Frank Hutter},
url = {https://doi.org/10.1613/jair.1.13922
https://github.com/automl/2022_JAIR_DAC_experiments, code},
year = {2022},
date = {2022-12-30},
urldate = {2022-05-30},
journal = {Journal of Artificial Intelligence Research (JAIR)},
volume = {75},
pages = {1633-1699},
abstract = {The performance of an algorithm often critically depends on its parameter configuration. While a variety of automated algorithm configuration methods have been proposed to relieve users from the tedious and error-prone task of manually tuning parameters, there is still a lot of untapped potential as the learned configuration is static, i.e., parameter settings remain fixed throughout the run. However, it has been shown that some algorithm parameters are best adjusted dynamically during execution. Thus far, this is most commonly achieved through hand-crafted heuristics. A promising recent alternative is to automatically learn such dynamic parameter adaptation policies from data. In this article, we give the first comprehensive account of this new field of automated dynamic algorithm configuration (DAC), present a series of recent advances, and provide a solid foundation for future research in this field. Specifically, we (i) situate DAC in the broader historical context of AI research; (ii) formalize DAC as a computational problem; (iii) identify the methods used in prior art to tackle this problem; and (iv) conduct empirical case studies for using DAC in evolutionary optimization, AI planning, and machine learning.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
算法的性能常常取决于其参数配置。虽然已经提出了各种自动化算法配置方法来减轻用户从手动调整参数的繁琐且容易出错的任务中解脱出来，但由于学习到的配置是静态的（即，参数设置在运行期间保持固定），因此仍有许多未被充分利用的潜力。然而，已有研究表明，一些算法参数在执行期间动态调整是最佳的。迄今为止，这最常通过手工设计的启发式方法来实现。一个有前景的近期替代方法是从数据中自动学习这种动态参数自适应策略。在本文中，我们对这个新的自动化动态算法配置（DAC）领域进行了第一次全面的概述，介绍了一系列近期进展，并为该领域的未来研究奠定了坚实的基础。具体来说，我们（i）将 DAC 置于更广泛的人工智能研究历史背景中；（ii）将 DAC 正式化为一个计算问题；（iii）识别了以往研究中用于解决该问题的方法；（iv）并进行了实证案例研究，将 DAC 应用于进化优化、人工智能规划和机器学习。
38.
Xue, Ke; Xu, Jiacheng; Yuan, Lei; Li, Miqing; Qian, Chao; Zhang, Zongzhang; Yu, Yang
Multi-agent Dynamic Algorithm Configuration 会议文章
In: Proceedings of the 36th International Conference on Advances in Neural Information Processing Systems (NeurIPS'22), 2022。
@inproceedings{xue-neurips22a,
title = {Multi-agent Dynamic Algorithm Configuration},
author = {Ke Xue and Jiacheng Xu and Lei Yuan and Miqing Li and Chao Qian and Zongzhang Zhang and Yang Yu},
url = {https://arxiv.org/abs/2210.06835v1, arxiv},
year = {2022},
date = {2022-11-28},
urldate = {2022-11-28},
booktitle = {Proceedings of the 36th International Conference on Advances in Neural Information Processing Systems (NeurIPS'22)},
abstract = {Automated algorithm configuration relieves users from tedious, trial-and-error tuning tasks. A popular algorithm configuration tuning paradigm is dynamic algorithm configuration (DAC), in which an agent learns dynamic configuration policies across instances by reinforcement learning (RL). However, in many complex algorithms, there may exist different types of configuration hyperparameters, and such heterogeneity may bring difficulties for classic DAC which uses a single-agent RL policy. In this paper, we aim to address this issue and propose multi-agent DAC (MA-DAC), with one agent working for one type of configuration hyperparameter. MA-DAC formulates the dynamic configuration of a complex algorithm with multiple types of hyperparameters as a contextual multi-agent Markov decision process and solves it by a cooperative multi-agent RL (MARL) algorithm. To instantiate, we apply MA-DAC to a well-known optimization algorithm for multi-objective optimization problems. Experimental results show the effectiveness of MA-DAC in not only achieving superior performance compared with other configuration tuning approaches based on heuristic rules, multi-armed bandits, and single-agent RL, but also being capable of generalizing to different problem classes. Furthermore, we release the environments in this paper as a benchmark for testing MARL algorithms, with the hope of facilitating the application of MARL.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
自动化算法配置使研究人员免于繁琐的试错式调优任务。一种流行的算法配置调优范式是动态算法配置（DAC），其中代理通过强化学习（RL）学习跨实例的动态配置策略。然而，在许多复杂的算法中，可能存在不同类型的配置超参数，并且这种异质性可能给使用单一代理 RL 策略的经典 DAC 带来困难。在本文中，我们旨在解决这个问题，并提出多代理 DAC (MA-DAC)，其中一个代理处理一种类型的配置超参数。MA-DAC 将具有多种类型超参数的复杂算法的动态配置公式化为上下文多代理马尔可夫决策过程，并通过合作多代理 RL (MARL) 算法进行求解。为了具体说明，我们将 MA-DAC 应用于解决多目标优化问题的著名优化算法。实验结果表明，MA-DAC 不仅在与基于启发式规则、多臂老虎机和单一代理 RL 的其他配置调优方法相比时表现出更优越的性能，而且还能够泛化到不同的问题类别。此外，我们在本文中发布了这些环境作为测试 MARL 算法的基准，希望促进 MARL 的应用。
37.
Biedenkapp, André
Dynamic Algorithm Configuration by Reinforcement Learning 博士论文
2022.
@phdthesis{biedenkapp22,
title = {Dynamic Algorithm Configuration by Reinforcement Learning},
author = {André Biedenkapp},
url = {https://freidok.uni-freiburg.de/data/230869, UniFreiburg
https://ml.informatik.uni-freiburg.de/wp-content/uploads/2022/11/2022_Dissertation_Andre_Biedenkapp.pdf, pdf},
year = {2022},
date = {2022-10-14},
abstract = {The performance of algorithms, be it in the domain of machine learning, hard combinatorial problem solving or AI in general depends on their many parameters. Tuning an algorithm manually, however, is error-prone and very time-consuming. Many, if not most, algorithms are iterative in nature. Thus, they traverse a potentially diverse solution space, which might require different parameter settings at different stages to behave optimally. Further, algorithms are often used for solving a diverse set of problem instances, which by themselves might require different parameters. Taking all of this into account is infeasible for a human designer. Automated methods have therefore been proposed to mitigate human errors and minimize manual efforts. While such meta-algorithmic methods have shown large successes, there is still a lot of untapped potentials as prior approaches typically only consider configurations that do not change during an algorithm’s run or do not adapt to the problem instance.
In this dissertation, we present the first framework that is capable of dynamically configuring algorithms, in other words, capable of adapting configurations to the problem instance at hand during an algorithm’s solving process. To this end, we model the dynamic algorithm configuration (DAC) problem as a contextual Markov decision process. This enables us to learn dynamic configuration policies in a data-driven way by means of reinforcement learning.
We empirically demonstrate the effectiveness of our framework on a diverse set of problem settings consisting of artificial benchmarks, evolutionary algorithms, AI planning systems, as well as deep learning. We show that DAC outperforms previous meta-algorithmic approaches. Building on these successes, we formulate the first standardized interface for dynamic configuration and an extensive benchmark to facilitate reproducibility and lower the barrier of entry for new researchers into this novel research field. Lastly, our work on DAC feeds back into the reinforcement learning paradigm. Through the lens of DAC, we identify shortcomings in current state-of-the-art approaches and demonstrate how to solve these. In particular, intending to learn general policies for DAC, our work pushes the boundaries of generalization in reinforcement learning. We demonstrate how to efficiently incorporate domain knowledge when training general agents and propose to move from a reactive way of doing reinforcement learning to a proactive way by learning when to make new decisions.},
howpublished = {https://freidok.uni-freiburg.de/data/230869},
keywords = {},
pubstate = {published},
tppubtype = {phdthesis}
}
算法的性能，无论是在机器学习、硬组合问题求解还是人工智能领域，都取决于其众多参数。然而，手动调整算法既容易出错，又非常耗时。许多，如果不是大多数，算法本质上是迭代的。因此，它们遍历一个潜在的多元解空间，这可能需要在不同阶段进行不同的参数设置才能表现最佳。此外，算法通常用于解决各种不同的问题实例，这些实例本身可能需要不同的参数。人类设计者不可能考虑所有这些因素。因此，已经提出了自动化方法来减轻人为错误并最大限度地减少手动工作。尽管此类元算法方法取得了巨大成功，但由于先前的方法通常只考虑在算法运行期间不变的配置或不适应问题实例，因此仍有许多未被充分利用的潜力。
在本论文中，我们提出了第一个能够动态配置算法的框架，换句话说，能够动态地根据算法求解过程中的问题实例调整配置。为此，我们将动态算法配置（DAC）问题建模为上下文马尔可夫决策过程。这使得我们能够通过强化学习以数据驱动的方式学习动态配置策略。
我们通过实验证明了我们的框架在各种问题设置下的有效性，包括人工基准、进化算法、人工智能规划系统以及深度学习。我们表明 DAC 的性能优于先前的元算法方法。在此基础上，我们提出了第一个用于动态配置的标准接口以及一个广泛的基准，以促进可复现性并降低新研究人员进入这个新研究领域的门槛。最后，我们关于 DAC 的工作反馈到强化学习范式中。通过 DAC 的视角，我们识别了当前最先进方法的不足之处，并展示了如何解决这些问题。特别是，为了学习 DAC 的通用策略，我们的工作突破了强化学习泛化的界限。我们展示了如何有效地在训练通用代理中融入领域知识，并提出了从反应式强化学习转向主动式强化学习，即学习何时做出新决策。
在本论文中，我们提出了第一个能够动态配置算法的框架，换句话说，能够动态地根据算法求解过程中的问题实例调整配置。为此，我们将动态算法配置（DAC）问题建模为上下文马尔可夫决策过程。这使得我们能够通过强化学习以数据驱动的方式学习动态配置策略。
我们通过实验证明了我们的框架在各种问题设置下的有效性，包括人工基准、进化算法、人工智能规划系统以及深度学习。我们表明 DAC 的性能优于先前的元算法方法。在此基础上，我们提出了第一个用于动态配置的标准接口以及一个广泛的基准，以促进可复现性并降低新研究人员进入这个新研究领域的门槛。最后，我们关于 DAC 的工作反馈到强化学习范式中。通过 DAC 的视角，我们识别了当前最先进方法的不足之处，并展示了如何解决这些问题。特别是，为了学习 DAC 的通用策略，我们的工作突破了强化学习泛化的界限。我们展示了如何有效地在训练通用代理中融入领域知识，并提出了从反应式强化学习转向主动式强化学习，即学习何时做出新决策。
36.
米歇尔特萨里，乔瓦尼雅卡
Reinforcement learning based adaptive metaheuristics 研讨会
Genetic and Evolutionary Computation Conference (GECCO) 2022, Companion Proceedings, 2022。
@workshop{tessari-geccocompanion22a,
title = {Reinforcement learning based adaptive metaheuristics},
author = {Michele Tessari, Giovanni Iacca},
url = {https://arxiv.org/abs/2206.12233, arxiv},
year = {2022},
date = {2022-07-10},
urldate = {2022-07-10},
booktitle = {Genetic and Evolutionary Computation Conference (GECCO) 2022, Companion Proceedings},
abstract = {Parameter adaptation, that is the capability to automatically adjust an algorithm's hyperparameters depending on the problem being faced, is one of the main trends in evolutionary computation applied to numerical optimization. While several handcrafted adaptation policies have been proposed over the years to address this problem, only few attempts have been done so far at applying machine learning to learn such policies. Here, we introduce a general-purpose framework for performing parameter adaptation in continuous-domain metaheuristics based on state-of-the-art reinforcement learning algorithms. We demonstrate the applicability of this framework on two algorithms, namely Covariance Matrix Adaptation Evolution Strategies (CMA-ES) and Differential Evolution (DE), for which we learn, respectively, adaptation policies for the step-size (for CMA-ES), and the scale factor and crossover rate (for DE). We train these policies on a set of 46 benchmark functions at different dimensionalities, with various inputs to the policies, in two settings: one policy per function, and one global policy for all functions. Compared, respectively, to the Cumulative Step-size Adaptation (CSA) policy and to two well-known adaptive DE variants (iDE and jDE), our policies are able to produce competitive results in the majority of cases, especially in the case of DE. },
keywords = {},
pubstate = {published},
tppubtype = {workshop}
}
参数自适应，即根据所面对的问题自动调整算法超参数的能力，是进化计算应用于数值优化中的主要趋势之一。尽管多年来已经提出了几种手工设计的自适应策略来解决这个问题，但到目前为止，将机器学习应用于学习这些策略的尝试还很少。在这里，我们基于最先进的强化学习算法，提出了一种用于连续域元启发式的参数自适应通用框架。我们通过在两种算法上展示该框架的应用，即协方差矩阵自适应进化策略（CMA-ES）和差分进化（DE），其中我们分别学习了步长（对于 CMA-ES）以及缩放因子和交叉率（对于 DE）的自适应策略。我们在 46 个不同维度的基准函数集上训练了这些策略，在两种设置下：每个函数一个策略，以及所有函数一个全局策略。与累积步长自适应（CSA）策略和两个知名的自适应 DE 变体（iDE 和 jDE）相比，我们的策略在大多数情况下能够产生具有竞争力的结果，尤其是在 DE 的情况下。
35.
Biedenkapp, André; Dang, Nguyen; Krejca, Martin S.; Hutter, Frank; Doerr, Carola
Theory-inspired Parameter Control Benchmarks for Dynamic Algorithm Configuration 会议文章
In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO'22), 2022。
@inproceedings{biedenkapp-gecco22a,
title = {Theory-inspired Parameter Control Benchmarks for Dynamic Algorithm Configuration},
author = {André Biedenkapp and Nguyen Dang and Martin S. Krejca and Frank Hutter and Carola Doerr},
url = {https://arxiv.org/abs/2202.03259},
year = {2022},
date = {2022-07-09},
urldate = {2022-07-09},
booktitle = {Proceedings of the Genetic and Evolutionary Computation Conference (GECCO'22)},
journal = {arXiv:2202.03259 [cs.NE]},
abstract = {It has long been observed that the performance of evolutionary algorithms and other randomized search heuristics can benefit from a non-static choice of the parameters that steer their optimization behavior. Mechanisms that identify suitable configurations on the fly ("parameter control") or via a dedicated training process ("dynamic algorithm configuration") are therefore an important component of modern evolutionary computation frameworks. Several approaches to address the dynamic parameter setting problem exist, but we barely understand which ones to prefer for which applications. As in classical benchmarking, problem collections with a known ground truth can offer very meaningful insights in this context. Unfortunately, settings with well-understood control policies are very rare.
One of the few exceptions for which we know which parameter settings minimize the expected runtime is the LeadingOnes problem. We extend this benchmark by analyzing optimal control policies that can select the parameters only from a given portfolio of possible values. This also allows us to compute optimal parameter portfolios of a given size. We demonstrate the usefulness of our benchmarks by analyzing the behavior of the DDQN reinforcement learning approach for dynamic algorithm configuration. },
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
人们早就观察到，进化算法和其他随机搜索启发式算法的性能可以通过非静态地选择引导其优化行为的参数来受益。因此，在需要时识别合适配置（“参数控制”）或通过专用训练过程（“动态算法配置”）的机制是现代进化计算框架的重要组成部分。存在几种解决动态参数设置问题的方法，但我们对哪些方法适用于哪些应用知之甚少。与经典基准测试一样，具有真实基础（ground truth）的问题集合可以提供非常有意义的见解。不幸的是，具有良好理解的控制策略的设置非常罕见。
少数例外之一是我们知道哪些参数设置可以最小化期望运行时间的，即 LeadingOnes 问题。我们通过分析可以在给定可能值组合中选择参数的最优控制策略来扩展此基准。这还允许我们计算给定大小的最优参数组合。我们通过分析 DDQN 强化学习方法在动态算法配置中的行为来证明我们基准的实用性。
少数例外之一是我们知道哪些参数设置可以最小化期望运行时间的，即 LeadingOnes 问题。我们通过分析可以在给定可能值组合中选择参数的最优控制策略来扩展此基准。这还允许我们计算给定大小的最优参数组合。我们通过分析 DDQN 强化学习方法在动态算法配置中的行为来证明我们基准的实用性。
34.
Biedenkapp, André; Speck, David; Sievers, Silvan; Hutter, Frank; Lindauer, Marius; Seipp, Jendrik
Learning Domain-Independent Policies for Open List Selection 研讨会
Workshop on Bridging the Gap Between AI Planning and Reinforcement Learning (PRL @ ICAPS'22), 2022。
@workshop{biedenkapp-prl22a,
title = {Learning Domain-Independent Policies for Open List Selection},
author = {André Biedenkapp and David Speck and Silvan Sievers and Frank Hutter and Marius Lindauer and Jendrik Seipp},
url = {http://ml.informatik.uni-freiburg.de/wp-content/uploads/2022/06/22-PRL-DAC4AIPlanning.pdf, pdf},
year = {2022},
date = {2022-06-13},
urldate = {2022-06-13},
booktitle = {Workshop on Bridging the Gap Between AI Planning and Reinforcement Learning (PRL @ ICAPS'22)},
abstract = {Since its proposal over a decade ago, LAMA has been con-
sidered one of the best-performing satisficing classical plan-
ners. Its key component is heuristic search with multiple open
lists, each using a different heuristic function to order states.
Even with a very simple, ad-hoc policy for open list selec-
tion, LAMA achieves state-of-the-art results. In this paper, we
propose to use dynamic algorithm configuration to learn such
policies in a principled and data-driven manner. On the learn-
ing side, we show how to train a reinforcement learning agent
over several heterogeneous environments, aiming at zero-shot
generalization to new related domains. On the planning side,
our experimental results show that the trained policies often
reach the performance of LAMA, and sometimes even per-
form better. Furthermore, our analysis of different policies
shows that prioritizing states reached via preferred operators
is crucial, explaining the strong performance of LAMA.},
keywords = {},
pubstate = {published},
tppubtype = {workshop}
}
自十多年前提出以来，LAMA 一直被认为是性能最佳的满足型经典规划器之一。其核心组件是具有多个开放列表的启发式搜索，每个列表使用不同的启发式函数来排序状态。即使采用非常简单、临时的开放列表选择策略，LAMA 也能取得最先进的结果。在本文中，我们提出使用动态算法配置来以原则性和数据驱动的方式学习此类策略。在学习方面，我们展示了如何通过几个异构环境来训练强化学习代理，目标是实现对新相关领域的零样本泛化。在规划方面，我们的实验结果表明，训练过的策略通常能达到 LAMA 的性能，有时甚至会优于 LAMA。此外，我们对不同策略的分析表明，优先考虑通过首选运算符到达的状态至关重要，这解释了 LAMA 的强大性能。
sidered one of the best-performing satisficing classical plan-
ners. Its key component is heuristic search with multiple open
lists, each using a different heuristic function to order states.
Even with a very simple, ad-hoc policy for open list selec-
tion, LAMA achieves state-of-the-art results. In this paper, we
propose to use dynamic algorithm configuration to learn such
policies in a principled and data-driven manner. On the learn-
ing side, we show how to train a reinforcement learning agent
over several heterogeneous environments, aiming at zero-shot
generalization to new related domains. On the planning side,
our experimental results show that the trained policies often
reach the performance of LAMA, and sometimes even per-
form better. Furthermore, our analysis of different policies
shows that prioritizing states reached via preferred operators
is crucial, explaining the strong performance of LAMA.
sidered one of the best-performing satisficing classical plan-
ners. Its key component is heuristic search with multiple open
lists, each using a different heuristic function to order states.
Even with a very simple, ad-hoc policy for open list selec-
tion, LAMA achieves state-of-the-art results. In this paper, we
propose to use dynamic algorithm configuration to learn such
policies in a principled and data-driven manner. On the learn-
ing side, we show how to train a reinforcement learning agent
over several heterogeneous environments, aiming at zero-shot
generalization to new related domains. On the planning side,
our experimental results show that the trained policies often
reach the performance of LAMA, and sometimes even per-
form better. Furthermore, our analysis of different policies
shows that prioritizing states reached via preferred operators
is crucial, explaining the strong performance of LAMA.
33.
Bhatia, Abhinav; Svegliato, Justin; Nashed, Samer B.; Zilberstein, Shlomo
In: Proceedings of the 32nd International Conference on Automated Planning and Scheduling (ICAPS'22), 2022。
@inproceedings{bathia-icaps22a,
title = {Tuning the Hyperparameters of Anytime Planning:A Metareasoning Approach with Deep Reinforcement Learning},
author = {Abhinav Bhatia and Justin Svegliato and Samer B. Nashed and Shlomo Zilberstein },
url = {https://ojs.aaai.org/index.php/ICAPS/article/view/19842/19601},
year = {2022},
date = {2022-06-13},
urldate = {2022-06-13},
booktitle = {Proceedings of the 32nd International Conference on Automated Planning and Scheduling (ICAPS'22)},
abstract = {Anytime planning algorithms often have hyperparameters that can be tuned at runtime to optimize their performance. While work on metareasoning has focused on when to interrupt an anytime planner and act on the current plan, the scope of metareasoning can be expanded to tuning the hyperparameters of the anytime planner at runtime. This paper introduces a general, decision-theoretic metareasoning approach that optimizes both the stopping point and hyperparameters of anytime planning. We begin by proposing a generalization of the standard meta-level control problem for anytime algorithms. We then offer a meta-level control technique that monitors and controls an anytime algorithm using deep reinforcement learning. Finally, we show that our approach boosts performance on a common benchmark domain that uses anytime weighted A* to solve a range of heuristic search problems and a mobile robot application that uses RRT* to solve motion planning problems. },
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
任何时间规划算法通常都有超参数可以在运行时进行调整以优化其性能。虽然关于元推理（metareasoning）的工作一直关注何时中断任何时间规划器并根据当前规划采取行动，但元推理的范围可以扩展到运行时调整任何时间规划器的超参数。本文介绍了一种通用的、基于决策论的元推理方法，该方法同时优化任何时间规划的停止点和超参数。我们首先提出对任何时间算法的标准元控制问题的推广。然后，我们提供一种元控制技术，该技术使用深度强化学习来监视和控制任何时间算法。最后，我们表明我们的方法在一种常用的基准领域中得到了提升，该领域使用任何时间加权 A* 来解决一系列启发式搜索问题，并在使用 RRT* 解决运动规划问题的移动机器人应用中得到了提升。
32.
Mandhane, Amol; Zhernov, Anton; Rauh, Maribeth; Gu, Chenjie; Wang, Miaosen; Xue, Flora; Shang, Wendy; Pang, Derek; Claus, Rene; Chiang, Ching-Han; others,
MuZero with self-competition for rate control in vp9 video compression 未发表
2022.
@unpublished{mandhane-arxiv22a,
title = {Muzero with self-competition for rate control in vp9 video compression},
author = {Amol Mandhane and Anton Zhernov and Maribeth Rauh and Chenjie Gu and Miaosen Wang and Flora Xue and Wendy Shang and Derek Pang and Rene Claus and Ching-Han Chiang and others},
url = {https://arxiv.org/abs/2202.06626},
year = {2022},
date = {2022-02-14},
urldate = {2023-06-13},
abstract = {Video streaming usage has seen a significant rise as entertainment, education, and business increasingly rely on online video. Optimizing video compression has the potential to increase access and quality of content to users, and reduce energy use and costs overall. In this paper, we present an application of the MuZero algorithm to the challenge of video compression. Specifically, we target the problem of learning a rate control policy to select the quantization parameters (QP) in the encoding process of libvpx, an open source VP9 video compression library widely used by popular video-on-demand (VOD) services. We treat this as a sequential decision making problem to maximize the video quality with an episodic constraint imposed by the target bitrate. Notably, we introduce a novel self-competition based reward mechanism to solve constrained RL with variable constraint satisfaction difficulty, which is challenging for existing constrained RL methods. We demonstrate that the MuZero-based rate control achieves an average 6.28% reduction in size of the compressed videos for the same delivered video quality level (measured as PSNR BD-rate) compared to libvpx's two-pass VBR rate control policy, while having better constraint satisfaction behavior.},
keywords = {},
pubstate = {published},
tppubtype = {unpublished}
}
视频流使用量大幅上升，因为娱乐、教育和商业越来越依赖在线视频。优化视频压缩可以提高用户访问和内容质量，并总体上降低能源消耗和成本。在本文中，我们将 MuZero 算法应用于视频压缩挑战。具体来说，我们针对学习率控制策略的问题，在 libvpx 的编码过程中选择量化参数（QP），libvpx 是一个被流行的视频点播（VOD）服务广泛使用的开源 VP9 视频压缩库。我们将此视为一个序贯决策问题，旨在在目标比特率的周期性约束下最大化视频质量。值得注意的是，我们引入了一种新颖的基于自我竞争的奖励机制来解决具有可变约束满足难度的约束强化学习问题，这对于现有的约束强化学习方法来说是一个挑战。我们证明，与 libvpx 的两通道 VBR 率控制策略相比，基于 MuZero 的率控制在相同的视频质量水平（以 PSNR BD-rate 衡量）下，平均可将压缩视频的大小减少 6.28%，同时具有更好的约束满足行为。
2021
31.
Getzelman, Grant; Balaprakash, Prasanna
Learning to Switch Optimizers for Quadratic Programming 会议文章
In: Balasubramanian, Vineeth N.; Tsang, Ivor (Ed.): Proceedings of The 13th Asian Conference on Machine Learning, pp. 1553–1568, PMLR, 2021。
@inproceedings{getzelman-acml21a,
title = {Learning to Switch Optimizers for Quadratic Programming},
author = {Getzelman, Grant and Balaprakash, Prasanna},
editor = {Balasubramanian, Vineeth N. and Tsang, Ivor},
url = {https://proceedings.mlr.press/v157/getzelman21a.html},
year = {2021},
date = {2021-11-17},
booktitle = {Proceedings of The 13th Asian Conference on Machine Learning},
volume = {157},
pages = {1553--1568},
publisher = {PMLR},
series = {Proceedings of Machine Learning Research},
abstract = {Quadratic programming (QP) seeks to solve optimization problems involving quadratic functions that can include complex boundary constraints. QP in the unrestricted form is $mathcal{NP}$-hard; but when restricted to the convex case, it becomes tractable. Active set and interior point methods are used to solve convex problems, and in the nonconvex case various heuristics or relaxations are used to produce high-quality solutions in finite time. Learning to optimize (L2O) is an emerging approach to design solvers for optimization problems. We develop an L2O approach that uses reinforcement learning to learn a stochastic policy to switch between pre-existing optimization algorithms to solve QP problem instances. In particular, our agent switches between three simple optimizers: Adam, gradient descent, and random search. Our experiments show that the learned optimizer minimizes quadratic functions faster and finds better-quality solutions in the long term than do any of the possible optimizers switched between. We also compare our solver with the standard QP algorithms in MATLAB and find better performance in fewer function evaluations.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
二次规划 (QP) 旨在解决涉及二次函数且可能包含复杂边界约束的优化问题。无约束形式的 QP 是 NP-hard 的；但当限制为凸时，它就变得易于处理。活动集和内点法用于解决凸问题，而在非凸情况下，则使用各种启发式或松弛方法来在有限时间内产生高质量的解决方案。学习优化 (L2O) 是一种设计优化问题求解器的新兴方法。我们开发了一种 L2O 方法，该方法使用强化学习来学习一个随机策略，以在预先存在的优化算法之间切换以解决 QP 问题实例。特别是，我们的代理在三种简单的优化器之间切换：Adam、梯度下降和随机搜索。我们的实验表明，学习到的优化器比任何可切换的优化器都能更快地最小化二次函数，并在长期内找到更高质量的解决方案。我们还将我们的求解器与 MATLAB 中的标准 QP 算法进行了比较，发现在较少函数评估的情况下性能更好。
30.
奥列戈维奇马拉申罗曼
Sparsely Ensembled Convolutional Neural Network Classifiers via Reinforcement Learning 会议文章
In: 2021 6th International Conference on Machine Learning Technologies, pp. 102–110, 2021, ISBN: 9781450389402。
@inproceedings{malashin-acm21,
title = {Sparsely Ensembled Convolutional Neural Network Classifiers via Reinforcement Learning},
author = {Olegovich Malashin, Roman},
url = {https://dl.acm.org/doi/10.1145/3468891.3468906
https://arxiv.org/abs/2102.03921},
doi = {10.1145/3468891.3468906},
isbn = {9781450389402},
year = {2021},
date = {2021-09-06},
booktitle = {2021 6th International Conference on Machine Learning Technologies},
pages = {102–110},
series = {ICMLT 2021},
abstract = {We consider convolutional neural network (CNN) ensemble learning with the objective function inspired by the least action principle; it includes resource consumption component. We teach an agent to perceive images through the set of pre-trained classifiers and want the resulting dynamically configured system to unfold the computational graph with the trajectory that refers to the minimal number of operations and maximal expected accuracy. The proposed agent's architecture implicitly approximates the required classifier selection function with the help of reinforcement learning. Our experimental results prove, that if the agent exploits the dynamic (and context-dependent) structure of computations, it outperforms conventional ensemble learning.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
我们考虑卷积神经网络 (CNN) 集成学习，其目标函数受最小作用量原理的启发；它包含资源消耗部分。我们训练一个代理通过一组预训练的分类器来感知图像，并希望由此产生的动态配置系统展开计算图，其轨迹指的是最少的操作和最大的预期准确率。所提出的代理架构利用强化学习来隐式地逼近所需的分类器选择函数。我们的实验结果证明，如果代理利用计算的动态（和上下文相关）结构，它将优于传统的集成学习。
29.
Nguyen, Manh Hung; Grinsztajn, Nathan; Guyon, Isabelle; Sun-Hosoy, Lisheng
MetaREVEAL: RL-based Meta-learning from Learning Curves 会议文章
In: Workshop on Interactive Adaptive Learning co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2021), 2021。
@inproceedings{nguyen-ial21a,
title = {MetaREVEAL: RL-based Meta-learning from Learning Curves},
author = {Manh Hung Nguyen and Nathan Grinsztajn and Isabelle Guyon and Lisheng Sun-Hosoy},
url = {https://hal.inria.fr/hal-03502358v2/document},
year = {2021},
date = {2021-09-01},
urldate = {2021-09-01},
booktitle = {Workshop on Interactive Adaptive Learning co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2021)},
abstract = {This paper addresses a cornerstone of Automated Machine Learning: the problem of rapidly uncovering which machine learning algorithm performs best on a new dataset. Our approach leverages performances of such algorithms on datasets to which they have been previously exposed, i.e., implementing a form of meta-learning. More specifically, the problem is cast as a REVEAL Reinforcement Learning (RL) game: the meta-learning problem is wrapped into a RL environment in which an agent can start, pause, or resume training various machine learning algorithms to progressively “reveal” their learning curves. The learned policy is then applied to quickly uncover the best algorithm on a new dataset. While other similar approaches, such as Freeze-Thaw, were proposed in the past, using Bayesian optimization, our methodology is, to the best of our knowledge, the first that trains a RL agent to do this task on previous datasets. Using real and artificial data, we show that our new RL-based meta-learning paradigm outperforms Free-Thaw and other baseline methods, with respect to the Area under the Learning curve metric, a form of evaluation of Any-time learning (i.e., the capability of interrupting the algorithm at any time while obtaining good performance).},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
本文解决了自动化机器学习（Automated Machine Learning）的一个基石问题：快速找出在新的数据集上哪种机器学习算法性能最佳的问题。我们的方法利用算法在数据集上的表现，即实现一种元学习形式。更具体地说，该问题被构建为一个 REVEAL 强化学习（RL）游戏：元学习问题被包装在一个 RL 环境中，代理可以在其中启动、暂停或恢复训练各种机器学习算法，以逐渐“揭示”它们的学习曲线。然后，将学习到的策略应用于快速找出新数据集上性能最佳的算法。尽管过去提出了诸如 Freeze-Thaw 等类似方法，使用贝叶斯优化，但据我们所知，我们的方法是第一个训练 RL 代理来完成这项任务（基于先前的数据集）的方法。通过使用真实和人工数据，我们表明我们的新 RL 驱动的元学习范式在学习曲线下面积（Area under the Learning curve）指标（一种评估任何时间学习（Any-time learning）的能力的形式，即在任何时候中断算法都能获得良好性能）方面优于 Freeze-Thaw 和其他基线方法。
28.
Ichnowski, Jeffrey; Jain, Paras; Stellato, Bartolomeo; Banjac, Goran; Luo, Michael; Borrelli, Francesco; Gonzalez, Joseph E.; Stoica, Ion; Goldberg, Ken
Accelerating Quadratic Optimization with Reinforcement Learning 未发表
2021.
@unpublished{ichnowski-arxiv21a,
title = {Accelerating Quadratic Optimization with Reinforcement Learning},
author = {Jeffrey Ichnowski and Paras Jain and Bartolomeo Stellato and Goran Banjac and Michael Luo and Francesco Borrelli and Joseph E. Gonzalez and Ion Stoica and Ken Goldberg},
url = {https://arxiv.org/abs/2107.10847},
year = {2021},
date = {2021-07-22},
journal = {arXiv},
abstract = {First-order methods for quadratic optimization such as OSQP are widely used for large-scale machine learning and embedded optimal control, where many related problems must be rapidly solved. These methods face two persistent challenges: manual hyperparameter tuning and convergence time to high-accuracy solutions. To address these, we explore how Reinforcement Learning (RL) can learn a policy to tune parameters to accelerate convergence. In experiments with well-known QP benchmarks we find that our RL policy, RLQP, significantly outperforms state-of-the-art QP solvers by up to 3x. RLQP generalizes surprisingly well to previously unseen problems with varying dimension and structure from different applications, including the QPLIB, Netlib LP and Maros-Meszaros problems. Code for RLQP is available at this https URL. },
keywords = {},
pubstate = {published},
tppubtype = {unpublished}
}
用于二次优化的一阶方法，如 OSQP，在大型机器学习和嵌入式最优控制中被广泛使用，因为这些场景需要快速求解许多相关问题。这些方法面临两个持续的挑战：手动超参数调优和收敛到高精度解的时间。为了解决这些问题，我们探索了强化学习（RL）如何学习一个策略来调优参数以加速收敛。在具有知名 QP 基准的实验中，我们发现我们的 RL 策略 RLQP 比最先进的 QP 求解器快达 3 倍。RLQP 对来自不同应用的、具有不同维度和结构但先前未见过的问题（包括 QPLIB、Netlib LP 和 Maros-Meszaros 问题）具有惊人的泛化能力。RLQP 的代码可在本文的 URL 找到。
27.
Speck, D; Biedenkapp, A; Hutter, F; Mattmüller, R; Lindauer, M
Learning Heuristic Selection with Dynamic Algorithm Configuration 会议文章
In: Zhuo, H H; Yang, Q; Do, M; Goldman, R; Biundo, S; Katz, M (Ed.): Proceedings of the 31st International Conference on Automated Planning and Scheduling (ICAPS'21), pp. 597–605, AAAI, 2021。
@inproceedings{speck-icaps21b,
title = {Learning Heuristic Selection with Dynamic Algorithm Configuration},
author = {D Speck and A Biedenkapp and F Hutter and R Mattmüller and M Lindauer},
editor = {H H Zhuo and Q Yang and M Do and R Goldman and S Biundo and M Katz},
url = {https://ojs.aaai.org/index.php/ICAPS/article/view/16008},
year = {2021},
date = {2021-01-01},
booktitle = {Proceedings of the 31st International Conference on Automated Planning and Scheduling (ICAPS'21)},
pages = {597--605},
publisher = {AAAI},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
26.
Eimer, T; Biedenkapp, A; Reimer, M; Adriaensen, S; Hutter, F; Lindauer, M
DACBench: A Benchmark Library for Dynamic Algorithm Configuration 会议文章
In: Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI'21), ijcai.org, 2021。
@inproceedings{eimer-ijcai21,
title = {DACBench: A Benchmark Library for Dynamic Algorithm Configuration},
author = {T Eimer and A Biedenkapp and M Reimer and S Adriaensen and F Hutter and M Lindauer},
url = {https://ml.informatik.uni-freiburg.de/papers/21-IJCAI-DACBench.pdf},
year = {2021},
date = {2021-01-01},
booktitle = {Proceedings of the Thirtieth International Joint Conference on
Artificial Intelligence (IJCAI'21)},
publisher = {ijcai.org},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
25.
Almeida, Diogo; Winter, Clemens; Tang, Jie; Zaremba, Wojciech
A Generalizable Approach to Learning Optimizers 未发表
2021.
@unpublished{almeida-arxiv21,
title = {A Generalizable Approach to Learning Optimizers},
author = {Diogo Almeida and Clemens Winter and Jie Tang and Wojciech Zaremba},
url = {https://arxiv.org/pdf/2106.00958.pdf},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
journal = {arXiv preprint arXiv:2106.00958},
keywords = {},
pubstate = {published},
tppubtype = {unpublished}
}
24.
Bhatia, Abhinav; Svegliato, Justin; Zilberstein, Shlomo
Tuning the Hyperparameters of Anytime Planning: A Deep Reinforcement Learning Approach 会议文章
In: ICAPS 2021 Workshop on Heuristics and Search for Domain-independent Planning, 2021。
@inproceedings{<LineBreak>bhatia-icaps21,
title = {Tuning the Hyperparameters of Anytime Planning: A Deep Reinforcement Learning Approach},
author = {Abhinav Bhatia and Justin Svegliato and Shlomo Zilberstein},
url = {https://openreview.net/forum?id=c7hpFp_eRCo},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {ICAPS 2021 Workshop on Heuristics and Search for Domain-independent Planning},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
2020
23.
Biedenkapp, André; Bozkurt, H. Furkan; Eimer, Theresa; Hutter, Frank; Lindauer, Marius
Dynamic Algorithm Configuration: Foundation of a New Meta-Algorithmic Framework 会议
Proceedings of the Twenty-fourth European Conference on Artificial Intelligence (ECAI'20), 2020。
@conference{biedenka-ecai,
title = {Dynamic Algorithm Configuration: Foundation of a New Meta-Algorithmic Framework},
author = {André Biedenkapp and H. Furkan Bozkurt and Theresa Eimer and Frank Hutter and Marius Lindauer},
url = {http://ecai2020.eu/papers/1237_paper.pdf},
year = {2020},
date = {2020-08-29},
booktitle = {Proceedings of the Twenty-fourth European Conference on Artificial Intelligence (ECAI'20)},
abstract = {The performance of many algorithms in the fields of
hard combinatorial problem solving, machine learning or AI in general depends on parameter tuning. Automated methods have been
proposed to alleviate users from the tedious and error-prone task of manually searching for performance-optimized configurations across
a set of problem instances. However, there is still a lot of untapped
potential through adjusting an algorithm’s parameters online since
different parameter values can be optimal at different stages of the
algorithm. Prior work showed that reinforcement learning is an effective approach to learn policies for online adjustments of algorithm
parameters in a data-driven way. We extend that approach by formulating the resulting dynamic algorithm configuration as a contextual
MDP, such that RL not only learns a policy for a single instance, but across a set of instances. To lay the foundation for studying dynamic
algorithm configuration with RL in a controlled setting, we propose
white-box benchmarks covering major aspects that make dynamic algorithm configuration a hard problem in practice and study the performance of various types of configuration strategies for them. On
these white-box benchmarks, we show that (i) RL is a robust candidate for learning configuration policies, outperforming standard parameter optimization approaches, such as classical algorithm configuration; (ii) based on function approximation, RL agents can learn to
generalize to new types of instances; and (iii) self-paced learning can
substantially improve the performance by selecting a useful sequence
of training instances automatically.},
keywords = {},
pubstate = {published},
tppubtype = {conference}
}
在硬组合问题求解、机器学习或一般人工智能领域的许多算法的性能都依赖于参数调优。已经提出了自动化方法来减轻用户从手动搜索跨问题实例的性能优化配置的繁琐且容易出错的任务中解脱出来。然而，通过在线调整算法的参数仍然有很大的潜力，因为不同的参数值在算法的不同阶段可能是最佳的。先前的工作表明，强化学习是一种有效的方法，可以以数据驱动的方式学习用于在线调整算法参数的策略。我们通过将由此产生的动态算法配置公式化为上下文 MDP 来扩展该方法，以便 RL 不仅可以为单个实例学习策略，还可以跨一系列实例学习策略。为了在可控的环境中研究使用 RL 进行动态算法配置的基础，我们提出了涵盖实践中使动态算法配置变得困难的主要方面的白盒基准，并研究了各种配置策略的性能。在这些白盒基准上，我们表明（i）RL 是学习配置策略的一个稳健选择，其性能优于经典算法配置等标准参数优化方法；（ii）基于函数逼近，RL 代理可以学习泛化到新类型的实例；（iii）自步学习可以通过自动选择有用的训练实例序列来显着提高性能。
hard combinatorial problem solving, machine learning or AI in general depends on parameter tuning. Automated methods have been
proposed to alleviate users from the tedious and error-prone task of manually searching for performance-optimized configurations across
a set of problem instances. However, there is still a lot of untapped
potential through adjusting an algorithm’s parameters online since
different parameter values can be optimal at different stages of the
algorithm. Prior work showed that reinforcement learning is an effective approach to learn policies for online adjustments of algorithm
parameters in a data-driven way. We extend that approach by formulating the resulting dynamic algorithm configuration as a contextual
MDP, such that RL not only learns a policy for a single instance, but across a set of instances. To lay the foundation for studying dynamic
algorithm configuration with RL in a controlled setting, we propose
white-box benchmarks covering major aspects that make dynamic algorithm configuration a hard problem in practice and study the performance of various types of configuration strategies for them. On
these white-box benchmarks, we show that (i) RL is a robust candidate for learning configuration policies, outperforming standard parameter optimization approaches, such as classical algorithm configuration; (ii) based on function approximation, RL agents can learn to
generalize to new types of instances; and (iii) self-paced learning can
substantially improve the performance by selecting a useful sequence
of training instances automatically.
hard combinatorial problem solving, machine learning or AI in general depends on parameter tuning. Automated methods have been
proposed to alleviate users from the tedious and error-prone task of manually searching for performance-optimized configurations across
a set of problem instances. However, there is still a lot of untapped
potential through adjusting an algorithm’s parameters online since
different parameter values can be optimal at different stages of the
algorithm. Prior work showed that reinforcement learning is an effective approach to learn policies for online adjustments of algorithm
parameters in a data-driven way. We extend that approach by formulating the resulting dynamic algorithm configuration as a contextual
MDP, such that RL not only learns a policy for a single instance, but across a set of instances. To lay the foundation for studying dynamic
algorithm configuration with RL in a controlled setting, we propose
white-box benchmarks covering major aspects that make dynamic algorithm configuration a hard problem in practice and study the performance of various types of configuration strategies for them. On
these white-box benchmarks, we show that (i) RL is a robust candidate for learning configuration policies, outperforming standard parameter optimization approaches, such as classical algorithm configuration; (ii) based on function approximation, RL agents can learn to
generalize to new types of instances; and (iii) self-paced learning can
substantially improve the performance by selecting a useful sequence
of training instances automatically.
22.
Gomoluch, Pawel; Alrajeh, Dalal; Russo, Alessandra; Bucchiarone, Antonio
Learning Neural Search Policies for Classical Planning 会议文章
In: Proceedings of the International Conference on Automated Planning and Scheduling, pp. 522–530, 2020。
@inproceedings{gomoluch-icaps20b,
title = {Learning Neural Search Policies for Classical Planning},
author = {Pawel Gomoluch and Dalal Alrajeh and Alessandra Russo and Antonio Bucchiarone},
url = {https://icaps20.icaps-conference.org/paper191.html},
year = {2020},
date = {2020-01-01},
urldate = {2020-01-01},
booktitle = {Proceedings of the International Conference on Automated Planning and Scheduling},
volume = {30},
pages = {522--530},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
21.
Shala, G; Biedenkapp, A; Awad, N; Adriaensen, S; Lindauer, M; Hutter, F
Learning Step-Size Adaptation in CMA-ES 会议文章
In: Proceedings of the Sixteenth International Conference on Parallel Problem Solving from Nature (PPSN'20), pp. 691–706, Springer, 2020。
@inproceedings{shala-ppsn20b,
title = {Learning Step-Size Adaptation in CMA-ES},
author = {G Shala and A Biedenkapp and N Awad and S Adriaensen and M Lindauer and F Hutter},
url = {https://ml.informatik.uni-freiburg.de/papers/20-PPSN-LTO-CMA.pdf},
year = {2020},
date = {2020-01-01},
urldate = {2020-01-01},
booktitle = {Proceedings of the Sixteenth International Conference on Parallel Problem Solving from Nature (PPSN'20)},
pages = {691--706},
publisher = {Springer},
series = {Lecture Notes in Computer Science},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
20.
Sae-Dan, Weerapan; Kessaci, Marie-Eléonore; Veerapen, Nadarajen; Jourdan, Laetitia
Time-Dependent Automatic Parameter Configuration of a Local Search Algorithm 会议文章
In: Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion, pp. 1898–1905, Association for Computing Machinery, Cancún, Mexico, 2020, ISBN: 9781450371278。
@inproceedings{sae-dan-gecco20b,
title = {Time-Dependent Automatic Parameter Configuration of a Local Search Algorithm},
author = {Weerapan Sae-Dan and Marie-Eléonore Kessaci and Nadarajen Veerapen and Laetitia Jourdan},
url = {https://doi.org/10.1145/3377929.3398107},
doi = {10.1145/3377929.3398107},
isbn = {9781450371278},
year = {2020},
date = {2020-01-01},
booktitle = {Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion},
pages = {1898–1905},
publisher = {Association for Computing Machinery},
address = {Cancún, Mexico},
series = {GECCO '20},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
2019
19.
Xu, Z; Dai, A M; Kemp, J; Metz, L
Learning an Adaptive Learning Rate Schedule 未发表
2019, (textitarXiv:1909.09712 [cs.LG])。
@unpublished{xu-arxiv19b,
title = {Learning an Adaptive Learning Rate Schedule},
author = {Z Xu and A M Dai and J Kemp and L Metz},
url = {https://arxiv.org/abs/1909.09712},
year = {2019},
date = {2019-01-01},
urldate = {2019-01-01},
note = {textitarXiv:1909.09712 [cs.LG]},
keywords = {},
pubstate = {published},
tppubtype = {unpublished}
}
18.
Sharma, Mudita; Komninos, Alexandros; nez, Manuel López-Ibá; Kazakov, Dimitar
Deep reinforcement learning based parameter control in differential evolution 会议文章
In: Auger, A; ü, St T (Ed.): Proceedings of the Genetic and Evolutionary Computation Conference (GECCO'19), pp. 709–717, ACM, 2019。
@inproceedings{sharma-gecco19b,
title = {Deep reinforcement learning based parameter control in differential evolution},
author = {Mudita Sharma and Alexandros Komninos and Manuel López-Ibá{~n}ez and Dimitar Kazakov},
editor = {A Auger and St T ü},
url = {https://dl.acm.org/doi/10.1145/3321707.3321813},
year = {2019},
date = {2019-01-01},
urldate = {2019-01-01},
booktitle = {Proceedings of the Genetic and Evolutionary Computation Conference (GECCO'19)},
pages = {709--717},
publisher = {ACM},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
17.
Gomoluch, Paweł; Alrajeh, Dalal; Russo, Alessandra
Learning classical planning strategies with policy gradient 会议文章
In: Proceedings of the International Conference on Automated Planning and Scheduling, pp. 637–645, 2019。
@inproceedings{gomoluch-icaps19b,
title = {Learning classical planning strategies with policy gradient},
author = {Pawe{ł} Gomoluch and Dalal Alrajeh and Alessandra Russo},
url = {https://ojs.aaai.org/index.php/ICAPS/article/view/3531},
year = {2019},
date = {2019-01-01},
urldate = {2019-01-01},
booktitle = {Proceedings of the International Conference on Automated Planning and Scheduling},
volume = {29},
pages = {637--645},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
2017
16.
Ansótegui, Carlos; Pon, Josep; Sellmann, Meinolf; Tierney, Kevin
Reactive Dialectic Search Portfolios for MaxSAT 会议文章
In: S.Singh,; Markovitch, S (Ed.): Proceedings of the Conference on Artificial Intelligence (AAAI'17), pp. 765–772, AAAI Press, 2017。
@inproceedings{ansotegui-aaai17b,
title = {Reactive Dialectic Search Portfolios for MaxSAT},
author = {Carlos Ansótegui and Josep Pon and Meinolf Sellmann and Kevin Tierney},
editor = {S.Singh and S Markovitch},
url = {https://ojs.aaai.org/index.php/AAAI/article/view/10660},
year = {2017},
date = {2017-01-01},
urldate = {2017-01-01},
booktitle = {Proceedings of the Conference on Artificial Intelligence (AAAI'17)},
pages = {765--772},
publisher = {AAAI Press},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
15.
Xu, Chang; Qin, Tao; Wang, Gang; Liu, Tie-Yan
Reinforcement learning for learning rate control 期刊文章
In: arXiv preprint arXiv:1705.11159, 2017。
@article{xu-arxiv17b,
title = {Reinforcement learning for learning rate control},
author = {Chang Xu and Tao Qin and Gang Wang and Tie-Yan Liu},
url = {https://arxiv.org/abs/1705.11159},
year = {2017},
date = {2017-01-01},
urldate = {2017-01-01},
journal = {arXiv preprint arXiv:1705.11159},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
14.
Kadioglu, S; Sellmann, M; Wagner, M
Learning a reactive restart strategy to improve stochastic search 会议文章
In: International Conference on Learning and Intelligent Optimization, pp. 109–123, Springer 2017。
@inproceedings{kadioglu-lion17b,
title = {Learning a reactive restart strategy to improve stochastic search},
author = {S Kadioglu and M Sellmann and M Wagner},
url = {https://cs.adelaide.edu.au/~markus/pub/2017lion-reactiveRestarts.pdf},
year = {2017},
date = {2017-01-01},
urldate = {2017-01-01},
booktitle = {International Conference on Learning and Intelligent Optimization},
pages = {109--123},
organization = {Springer},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
2016
13.
Adriaensen, S; Nowé, A
Towards a White Box Approach to Automated Algorithm Design 会议文章
In: Kambhampati, S (Ed.): Proceedings of the 26th International Joint Conference on Artificial Intelligence (IJCAI'16), pp. 554–560, 2016。
@inproceedings{adriaensen-ijcai16b,
title = {Towards a White Box Approach to Automated Algorithm Design},
author = {S Adriaensen and A Nowé},
editor = {S Kambhampati},
url = {https://www.ijcai.org/Proceedings/16/Papers/085.pdf},
year = {2016},
date = {2016-01-01},
urldate = {2016-01-01},
booktitle = {Proceedings of the 26th International Joint Conference on Artificial Intelligence (IJCAI'16)},
pages = {554--560},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
12.
汉森，萨曼莎
Using Deep Q-Learning to Control Optimization Hyperparameters 期刊文章
In: arXiv preprint arXiv:1602.04062, 2016。
@article{hansen-arxiv16b,
title = {Using Deep Q-Learning to Control Optimization Hyperparameters},
author = {Samantha Hansen},
url = {https://arxiv.org/pdf/1602.04062.pdf},
year = {2016},
date = {2016-01-01},
urldate = {2016-01-01},
journal = {arXiv preprint arXiv:1602.04062},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
11.
Andersson, Martin; Bandaru, Sunith; Ng, Amos HC
Tuning of Multiple Parameter Sets in Evolutionary Algorithms 会议文章
In: Proceedings of the Genetic and Evolutionary Computation Conference 2016, pp. 533–540, 2016。
@inproceedings{andersson-gecco16b,
title = {Tuning of Multiple Parameter Sets in Evolutionary Algorithms},
author = {Martin Andersson and Sunith Bandaru and Amos HC Ng},
url = {https://dl.acm.org/doi/10.1145/2908812.2908899},
year = {2016},
date = {2016-01-01},
urldate = {2016-01-01},
booktitle = {Proceedings of the Genetic and Evolutionary Computation Conference 2016},
pages = {533--540},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
10.
Daniel, C; Taylor, J; Nowozin, S
Learning Step Size Controllers for Robust Neural Network Training 会议文章
In: Schuurmans, D; Wellman, M (Ed.): Proceedings of the Thirtieth National Conference on Artificial Intelligence (AAAI'16), AAAI Press, 2016。
@inproceedings{daniel-aaai16b,
title = {Learning Step Size Controllers for Robust Neural Network Training},
author = {C Daniel and J Taylor and S Nowozin},
editor = {D Schuurmans and M Wellman},
url = {https://ojs.aaai.org/index.php/AAAI/article/view/10187},
year = {2016},
date = {2016-01-01},
urldate = {2016-01-01},
booktitle = {Proceedings of the Thirtieth National Conference on Artificial Intelligence (AAAI'16)},
publisher = {AAAI Press},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
2014
9.
López-Ibáñez, Manuel; Stützle, Thomas
Automatically Improving the Anytime Behaviour of Optimisation Algorithms 期刊文章
In: European Journal of Operational Research, vol. 235, no. 3, pp. 569–582, 2014。
@article{lopez-ejor14b,
title = {Automatically Improving the Anytime Behaviour of Optimisation Algorithms},
author = {Manuel López-Ibánez and Thomas Stützle},
url = {https://www.sciencedirect.com/science/article/abs/pii/S0377221713008667?via%3Dihub},
year = {2014},
date = {2014-01-01},
urldate = {2014-01-01},
journal = {European Journal of Operational Research},
volume = {235},
number = {3},
pages = {569--582},
publisher = {Elsevier},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2012
8.
Battiti, R; Campigotto, P
An Investigation of Reinforcement Learning for Reactive Search Optimization 书本章节
In: Hamadi, Y; Monfroy, E; Saubion, F (Ed.): Autonomous Search, pp. 131–160, Springer, 2012。
@incollection{battiti-as12ab,
title = {An Investigation of Reinforcement Learning for Reactive Search Optimization},
author = {R Battiti and P Campigotto},
editor = {Y Hamadi and E Monfroy and F Saubion},
url = {https://link.springer.com/chapter/10.1007%2F978-3-642-21434-9_6},
year = {2012},
date = {2012-01-01},
urldate = {2012-01-01},
booktitle = {Autonomous Search},
pages = {131--160},
publisher = {Springer},
keywords = {},
pubstate = {published},
tppubtype = {incollection}
}
2010
7.
Xu, Yuehua; Fern, Alan; Yoon, Sungwook
Iterative Learning of Weighted Rule Sets for Greedy Search 会议
Proceedings of the 20th International Conference on Automated Planning and Scheduling (ICAPS'10), 2010。
@conference{xu-icaps10a,
title = {Iterative Learning of Weighted Rule Sets for Greedy Search},
author = {Yuehua Xu and Alan Fern and Sungwook Yoon},
url = {https://www.aaai.org/ocs/index.php/ICAPS/ICAPS10/paper/view/1444},
year = {2010},
date = {2010-04-20},
booktitle = {Proceedings of the 20th International Conference on Automated Planning and Scheduling (ICAPS'10)},
abstract = {Greedy search is commonly used in an attempt to generate solutions quickly at the expense of completeness and optimality. In this work, we consider learning sets of weighted action-selection rules for guiding greedy search with application to automated planning. We make two primary contributions over prior work on learning for greedy search. First, we introduce weighted sets of action-selection rules as a new form of control knowledge for greedy search. Prior work has shown the utility of action-selection rules for greedy search, but has treated the rules as hard constraints, resulting in brittleness. Our weighted rule sets allow multiple rules to vote, helping to improve robustness to noisy rules. Second, we give a new iterative learning algorithm for learning weighted rule sets based on RankBoost, an efficient boosting algorithm for ranking. Each iteration considers the actual performance of the current rule set and directs learning based on the observed search errors. This is in contrast to most prior approaches, which learn control knowledge independently of the search process. Our empirical results have shown significant promise for this approach in a number of domains.},
keywords = {},
pubstate = {published},
tppubtype = {conference}
}
贪婪搜索通常用于尝试快速生成解决方案，但会牺牲完整性和最优性。在这项工作中，我们考虑学习一组加权的动作选择规则来指导贪婪搜索，并将其应用于自动规划。与以往学习贪婪搜索的控制知识的工作相比，我们做出了两项主要贡献。首先，我们将加权动作选择规则集作为一种新的形式引入了贪婪搜索的控制知识。以往的工作表明了动作选择规则对贪婪搜索的效用，但将规则视为硬约束，导致了脆性。我们的加权规则集允许多个规则投票，有助于提高对噪声规则的鲁棒性。其次，我们提供了一种新的迭代学习算法，该算法基于 RankBoost（一种高效的排序提升算法）来学习加权规则集。每次迭代都考虑当前规则集的实际性能，并根据观察到的搜索错误进行学习。这与大多数先前的方法不同，后者独立于搜索过程学习控制知识。我们的实证结果在该方法在几个领域显示出显著的希望。
6.
Sakurai, Y; Takada, K; Kawabe, T; Tsuruta, S
A Method to Control Parameters of Evolutionary Algorithms by Using Reinforcement Learning 会议文章
In: é, K Y; Dipanda, A; Chbeir, R (Ed.): Proceedings of Sixth International Conference on Signal-Image Technology and Internet-Based Systems (SITIS), pp. 74–79, IEEE Computer Society, 2010。
@inproceedings{sakurai-sitis10ab,
title = {A Method to Control Parameters of Evolutionary Algorithms by Using Reinforcement Learning},
author = {Y Sakurai and K Takada and T Kawabe and S Tsuruta},
editor = {K Y é and A Dipanda and R Chbeir},
url = {https://ieeexplore.ieee.org/document/5714532},
year = {2010},
date = {2010-01-01},
urldate = {2010-01-01},
booktitle = {Proceedings of Sixth International Conference on Signal-Image Technology and Internet-Based Systems (SITIS)},
pages = {74--79},
publisher = {IEEE Computer Society},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
5.
Fialho, Alvaro; Costa, Luis Da; Schoenauer, Marc; Sebag, Michele
Analyzing Bandit-Based Adaptive Operator Selection Mechanisms 期刊文章
In: Annals of Mathematics and Artificial Intelligence, vol. 60, no. 1, pp. 25–64, 2010。
@article{fialho-amai10b,
title = {Analyzing Bandit-Based Adaptive Operator Selection Mechanisms},
author = {Alvaro Fialho and Luis Da Costa and Marc Schoenauer and Michele Sebag},
url = {https://link.springer.com/article/10.1007%2Fs10472-010-9213-y},
year = {2010},
date = {2010-01-01},
urldate = {2010-01-01},
journal = {Annals of Mathematics and Artificial Intelligence},
volume = {60},
number = {1},
pages = {25--64},
publisher = {Springer},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2008
4.
Aine, Sandip; Kumar, Rajeev; Chakrabarti, P. P.
Adaptive parameter control of evolutionary algorithms to improve quality-time trade-off 期刊文章
In: Applied Soft Computing, vol. 9, no. 2, pp. 527-540, 2008。
@article{aine-jasoc08a,
title = {Adaptive parameter control of evolutionary algorithms to improve quality-time trade-off},
author = {Sandip Aine and Rajeev Kumar and P.P. Chakrabarti},
url = {https://doi.org/10.1016/j.asoc.2008.07.001},
year = {2008},
date = {2008-08-05},
urldate = {2008-08-05},
journal = {Applied Soft Computing},
volume = {9},
number = {2},
pages = {527-540},
abstract = {Parameter control of evolutionary algorithms (EAs) poses special challenges as EA uses a population and requires many parameters to be controlled for an effective search. Quality improvement is dependent on several factors, such as, fitness estimation, population diversity and convergence rate. A widely practiced approach to identify a good set of parameters for a particular class of problem is through experimentation. Ideally, the parameter selection should depend on the resource availability, and thus, a rigid choice may not be suitable. In this work, we propose an automated framework for parameter selection, which can adapt according to the constraints specified. To condition the parameter choice through resource constraint/utilization, we consider two typical scenarios, one where maximum available run-time is pre-specified and the other in which a utility function modeling the quality-time trade-off is used instead of a rigid deadline. We present static and dynamic parameter selection strategies based on a probabilistic profiling method. Experiments performed with traveling salesman problem (TSP) and standard cell placement problem show that an informed adaptive parameter control mechanism can yield better results than a static selection.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
参数控制演化算法 (EA) 带来了特殊的挑战，因为 EA 使用种群并需要控制许多参数才能进行有效的搜索。质量的提高取决于几个因素，例如适应度估计、种群多样性和收敛速度。识别特定问题类别的良好参数集的一种普遍实践方法是通过实验。理想情况下，参数选择应取决于资源可用性，因此，僵化的选择可能不合适。在这项工作中，我们提出了一个用于参数选择的自动化框架，该框架可以根据指定的约束进行调整。为了通过资源约束/利用来条件化参数选择，我们考虑了两种典型场景：一种是预先指定了最大可用运行时间，另一种是使用模拟质量-时间权衡的效用函数而不是僵化的截止日期。我们基于概率剖析方法提出了静态和动态参数选择策略。在旅行商问题 (TSP) 和标准单元放置问题上进行的实验表明，知情的自适应参数控制机制可以产生比静态选择更好的结果。
2002
3.
Pettinger, J; Everson, R
使用强化学习控制遗传算法 会议论文集文章
In: 第四届年度遗传与进化计算会议论文集, pp. 692–692, 2002.
@inproceedings{pettinger-gecco02b,
title = {Controlling Genetic Algorithms with Reinforcement Learning},
author = {J Pettinger and R Everson},
url = {https://dl.acm.org/doi/10.5555/2955491.2955607},
year = {2002},
date = {2002-01-01},
urldate = {2002-01-01},
booktitle = {Proceedings of the 4th Annual Conference on Genetic and Evolutionary Computation},
pages = {692--692},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
2001
2.
Lagoudakis, M; Littman, M
In: 离散数学电子笔记, vol. 9, pp. 344–359, 2001.
@article{lagoudakis-endm01ab,
title = {Learning to Select Branching Rules in the DPLL Procedure for Satisfiability},
author = {M Lagoudakis and M Littman},
url = {https://www.sciencedirect.com/science/article/abs/pii/S1571065304003324?via%3Dihub},
year = {2001},
date = {2001-01-01},
urldate = {2001-01-01},
journal = {Electronic Notes in Discrete Mathematics},
volume = {9},
pages = {344--359},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2000
1.
Lagoudakis, Michail G.; Littman, Michael L.
使用强化学习进行算法选择 会议
第 17 届国际机器学习会议 (ICML 2000) 论文集, 2000.
@conference{lagoudakis-icml2000a,
title = {Algorithm Selection using Reinforcement Learning},
author = { Michail G. Lagoudakis and Michael L. Littman },
url = {http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.472.7494},
year = {2000},
date = {2000-07-01},
booktitle = {Proceedings of the 17th International Conference on Machine Learning (ICML 2000)},
abstract = {Many computational problems can be solved by multiple algorithms, with different algorithms fastest for different problem sizes, input distributions, and hardware characteristics. We consider the problem of algorithm selection: dynamically choose an algorithm to attack an in-stance of a problem with the goal of minimiz-ing the overall execution time. We formulate the problem as a kind of Markov decision process (MDP), and use ideas from reinforcement learning to solve it. This paper introduces a kind of MDP that models the algorithm selection problem by allowing multiple state transitions. The well known Q-learning algorithm is adapted for this case in a way that combines both Monte-Carlo and Temporal Difference methods. Also, this work uses, and extends in a way to control problems, the Least-Squares Temporal Difference algorithm (LSTD(0)) of Boyan. The experimental study focuses on the classic problems of order statistic selection and sorting. The encouraging results reveal the potential of applying learning methods to traditional computational problems},
keywords = {},
pubstate = {published},
tppubtype = {conference}
}
许多计算问题可以通过多种算法解决，不同的算法在不同的问题大小、输入分布和硬件特性下速度更快。我们考虑算法选择问题：动态选择一种算法来处理问题的一个实例，目标是最小化总执行时间。我们将该问题形式化为一种马尔可夫决策过程 (MDP)，并使用强化学习的思想来解决它。本文介绍了一种模拟算法选择问题的 MDP，它允许多个状态转换。著名的 Q-learning 算法已在此类情况下进行改编，结合了蒙特卡罗和时间差分方法。此外，这项工作还使用了 Boyan 的最小二乘时间差分算法 (LSTD(0))，并对其进行了扩展以控制问题。实验研究侧重于顺序统计选择和排序的经典问题。令人鼓舞的结果揭示了将学习方法应用于传统计算问题的潜力。