Share via

What rank is used in LoRA fine-tuning?

Nick Matton 0 Reputation points Microsoft Employee
2025-01-11T20:47:25.69+00:00

I previously fine-tuned a few instances of gpt-4 using the standard Azure ML training flow. I am seeing online that Azure ML uses Low rank approximation in order to optimize fine tuning.

I am wondering what rank is used in the LoRA fine-tuning process so that I might determine the memory/flops saved by using Azure ML LoRA instead of running a full backprop on the model.

Thanks!

Azure Machine Learning
Azure Machine Learning
An Azure machine learning service for building and deploying models.
3,068 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Saideep Anchuri 935 Reputation points Microsoft Vendor
    2025-01-12T11:36:48.2233333+00:00

    Hi Nick Matton

    Welcome to Microsoft Q&A Forum, thank you for posting your query here!

    I understand that you are encountering an issue, The LoRA (Low-Rank Adaptation) fine-tuning process generally employs a rank between 4 and 32. This range ensures effective performance while greatly decreasing the number of trainable parameters and the overall memory requirements. Using LoRA (Low-Rank Adaptation) allows you to attain performance levels similar to those achieved with full fine-tuning.

    Kindly refer below link: huggingface

    Hope this helps. Do let us know if you any further queries.

     

    If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

    Thank You.

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.