What is Direct Preference Optimization (DPO)?<\/h2>\n
Direct Preference Optimization is an innovative technique for training language models to align with human preferences without the complexity of traditional Reinforcement Learning from Human Feedback (RLHF). Introduced in the groundbreaking paper\u00a0Direct Preference Optimization: Your Language Model is Secretly a Reward Model” by Rafailov, Sharma,…<\/a>, DPO fundamentally reimagines how we teach models to generate preferred outputs.<\/p>\n Unlike traditional supervised fine-tuning where you show a model “what to say,” DPO teaches models by showing them comparative examples: “this response is better than that one.” For each prompt, you provide:<\/p>\n The model learns to increase the likelihood of generating preferred responses while decreasing the probability of non-preferred ones, all without requiring explicit reward modeling or complex reinforcement learning pipelines.<\/p>\n Best Use Cases for DPO:<\/strong><\/p>\n The following code demonstrates DPO fine-tuning using the\u00a0Microsoft Foundry Projects SDK<\/a>:<\/p>\n DPO Fine-Tuning Results:<\/p>\n Inference result:<\/p>\n Data format example:<\/p>\n\n
![\"dpo](\"https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2026\/01\/dpo-image.webp\")
<\/a><\/p>\n\n
How Direct Preference Optimization Works<\/h2>\n
import os\r\nfrom dotenv import load_dotenv\r\nfrom azure.identity import DefaultAzureCredential\r\nfrom azure.ai.projects import AIProjectClient\r\n\r\n# Load environment variables\r\nload_dotenv()\r\n\r\nendpoint = os.environ.get(\"AZURE_AI_PROJECT_ENDPOINT\")\r\nmodel_name = os.environ.get(\"MODEL_NAME\")\r\n\r\n# Define dataset file paths\r\ntraining_file_path = \"training.jsonl\"\r\nvalidation_file_path = \"validation.jsonl\"\r\n\r\ncredential = DefaultAzureCredential()\r\nproject_client = AIProjectClient(endpoint=endpoint, credential=credential)\r\nopenai_client = project_client.get_openai_client()\r\n\r\n# Upload training and validation files\r\nwith open(training_file_path, \"rb\") as f:\r\n train_file = openai_client.files.create(file=f, purpose=\"fine-tune\")\r\n\r\nwith open(validation_file_path, \"rb\") as f:\r\n validation_file = openai_client.files.create(file=f, purpose=\"fine-tune\")\r\n\r\nopenai_client.files.wait_for_processing(train_file.id)\r\nopenai_client.files.wait_for_processing(validation_file.id)\r\n\r\n# Create DPO Fine Tuning job\r\nfine_tuning_job = openai_client.fine_tuning.jobs.create(\r\n training_file=train_file.id,\r\n validation_file=validation_file.id,\r\n model=model_name,\r\n method={\r\n \"type\": \"dpo\",\r\n \"dpo\": {\r\n \"hyperparameters\": {\r\n \"n_epochs\": 3,\r\n \"batch_size\": 1,\r\n \"learning_rate_multiplier\": 1.0\r\n }\r\n }\r\n },\r\n extra_body={\"trainingType\": \"GlobalStandard\"}\r\n)<\/code><\/pre>\nprint(f\"Testing fine-tuned model via deployment: {deployment_name}\")\r\n\r\nresponse = openai_client.responses.create(\r\n model=deployment_name,\r\n input=[{\"role\": \"user\", \"content\": \"Explain machine learning in simple terms.\"}]\r\n)\r\n\r\nprint(f\"Model response: {response.output_text}\")<\/code><\/pre>\nModel response: Machine learning is like teaching a computer to learn from experience, similar to how people do. Instead of programming specific instructions for every task, we give the computer a lot of data and it figures out patterns on its own. Then, it can use what it learned to make decisions or predictions. For example, if you show a machine learning system lots of pictures of cats and dogs, it will learn to recognize which is which by itself.<\/code><\/pre>\n{\r\n \"input\": {\r\n \"messages\": [\r\n {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\r\n {\"role\": \"user\", \"content\": \"What is the capital of France?\"}\r\n ]\r\n },\r\n \"preferred_output\": [\r\n {\"role\": \"assistant\", \"content\": \"The capital of France is Paris.\"}\r\n ],\r\n \"non_preferred_output\": [\r\n {\"role\": \"assistant\", \"content\": \"I think it's London.\"}\r\n ]\r\n}<\/code><\/pre>\nComparing DPO to Other Methods<\/h2>\n