Can machine learning fix its own problems? Hyperparameter tuning of machine learning with provable guarantees
Recent advances in machine learning have transformed the theory and practice of algorithm design and analysis. From a theoretical perspective, exciting new frameworks have emerged, such as data-driven algorithm design (learning domain-specific algorithms leveraging training instances from that domain) and algorithms with predictions (designing algorithms that leverage potentially imperfect machine learning predictions).
In this talk, we argue that we must also loop back and apply these new theoretical tools to optimize machine learning itself. I will present our work that builds on tools from data-driven algorithm design to provide provable guarantees for hyperparameter optimization of machine learning. Specifically, I will discuss two case studies: tuning algorithms for building decision trees (often favored in practice for their interpretability) and tuning of model hyperparameters in neural networks with structured parameter-dependent dual functions.