Seminar on Automatic Rank Determination for Low-Rank Adaptation via Submodular Function Maximisation
Abstract In this talk, we will introduce SubLoRA, a rank determination method for Low-Rank Adaptation (LoRA) based on submodular function maximisation. In contrast to prior approaches, such as AdaLoRA, that rely on first-order (linearised) approximations of the loss function, SubLoRA utilises second-order information to capture the potentially complex loss landscape by incorporating the Hessian matrix. […]
