How to build domain specific LLM from scratch

Step 1: Define Your Domain and Requirements

Before writing any code, clearly define your domain. Are you building for medical, legal, finance, or another specialized field? Understanding your scope determines everything from data sources to model architecture. Some important considerations to be kept in mind are:

• What specific problems will your LLM solve?
• Who are your end users?
• What level of accuracy do you need?

Domain-specific models excel because they focus exclusively on one subject area, eliminating the noise that comes with general-purpose models.

Step 2: Set Up Your Development Environment

You'll need proper hardware and software infrastructure. At minimum, you need a GPU-enabled system. Cloud options like AWS, GCP, or Azure work if you don't have local hardware.

Create your Python environment:

          
            # Create a virtual environment
            python -m venv llm-env
            source llm-env/bin/activate  # On Windows: llm-env\Scripts\activate

            # Install required packages
            pip install torch transformers datasets pandas numpy fastapi uvicorn
          
        

GPU:
- Fine-tuning small models (1–3B): RTX 3090 / 4090 (24GB VRAM)
- Medium models (7–13B): A100 40GB or multi-GPU setup

RAM:
- 32GB minimum (64GB recommended for large datasets)

Storage:
- 500GB–1TB SSD depending on dataset size

This makes our guide more realistic and approachable.

Step 3: Gather and Prepare Your Data

Data quality directly determines your model's performance. Start collecting domain-specific text from reliable sources.

Data sources include:

• Company documents and archives
• Industry databases
• Public datasets (government records, academic papers)
• Web scraping (with proper permissions)

Clean your data with Python:

          
            import pandas as pd
            import re

            def clean_text(text):
                # Convert to lowercase
                text = text.lower()
                # Remove special characters but keep domain-specific ones
                text = text.lower()
                text = re.sub(r'\s+', ' ', text)  # normalize whitespace
                return text.strip()

            # Load and clean your dataset
            df = pd.read_csv("raw_data.csv")
            df['cleaned_text'] = df['text'].apply(clean_text)
            df.to_csv("clean_data.csv", index=False)
          
        

Important: Don't over-clean. Preserve domain-specific syntax like chemical formulas or programming code.

Step 4: Split Your Dataset

Properly dividing your data ensures reliable model evaluation.

Ensure related documents (e.g., same case, patient, or contract) do not appear across multiple splits to prevent data leakage and inflated evaluation scores.

          
            from sklearn.model_selection import train_test_split

            # Split data: 70% train, 20% validation, 10% test
            train_data, temp_data = train_test_split(df, test_size=0.3, random_state=42)
            val_data, test_data = train_test_split(temp_data, test_size=0.33, random_state=42)

            # Save splits
            train_data.to_csv('train.csv', index=False)
            val_data.to_csv('validation.csv', index=False)
            test_data.to_csv('test.csv', index=False)
          
        

Step 5: Fine-Tune Your Model

Instead of training from scratch, fine-tune an existing model like GPT-2. This saves time and computational resources.

          
            from transformers import GPT2LMHeadModel, GPT2Tokenizer, Trainer, TrainingArguments
            from datasets import load_dataset

            # Load your prepared data
            dataset = load_dataset('text', data_files={
                'train': 'train.txt',
                'validation': 'val.txt'
            })

            # Initialize tokenizer and model
            tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
            model = GPT2LMHeadModel.from_pretrained('gpt2')

            # Tokenize your dataset
            def tokenize_function(examples):
                return tokenizer(
                    examples['text'],
                    truncation=True,
                    padding='max_length',
                    max_length=128
                )

            tokenized_dataset = dataset.map(tokenize_function, batched=True)

            # Configure training
            training_args = TrainingArguments(
                output_dir='./results',
                evaluation_strategy='epoch',
                num_train_epochs=3,
                per_device_train_batch_size=4,
                save_steps=500,
                logging_steps=100
            )

            # Train the model
            trainer = Trainer(
                model=model,
                args=training_args,
                train_dataset=tokenized_dataset['train'],
                eval_dataset=tokenized_dataset['validation']
            )

            trainer.train()

            # Save your fine-tuned model
            model.save_pretrained('./my-domain-model')
            tokenizer.save_pretrained('./my-domain-model')
          
        

Important parameters to adjust:

• Batch size: Start small (4-8) to avoid memory errors
• Learning rate: Use 1e-5 to 5e-5 for fine-tuning
• Epochs: Begin with 3-5, monitoring validation loss

Step 6: Create a Python Inference Server

Your model needs an API endpoint for generating responses. FastAPI provides a fast, modern solution.

          
            from fastapi import FastAPI, Body
            from transformers import GPT2LMHeadModel, GPT2Tokenizer
            import uvicorn

            app = FastAPI()

            # Load your trained model
            model_path = "./my-domain-model"
            tokenizer = GPT2Tokenizer.from_pretrained(model_path)
            model = GPT2LMHeadModel.from_pretrained(model_path)
            model.eval()

            @app.post("/generate")
            def generate_text(prompt_data: dict = Body(...)):
                prompt = prompt_data["prompt"]
                
                # Tokenize input
                inputs = tokenizer.encode(prompt, return_tensors='pt')
                
                # Generate response
                outputs = model.generate(
                    inputs,
                    max_length=100,
                    num_return_sequences=1,
                    do_sample=True,
                    top_p=0.9,
                    top_k=50
                )
                
                # Decode and return
                generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
                return {"response": generated_text}

            if __name__ == "__main__":
                uvicorn.run(app, host="0.0.0.0", port=5000)
          
        

Run your server: python inference_server.py

Step 7: Build a Java Middleware (Spring Boot)

For enterprise environments, add a secure middleware layer.

          
            // Example: Spring Boot Controller
            @RestController
            @RequestMapping("/api/v1/model")
            public class ModelController {
                
                @Autowired
                private ModelService modelService;
                
                @PostMapping("/generate")
                public ResponseEntity generateText(@RequestBody UserPrompt prompt) {
                    String response = modelService.getGeneratedText(prompt.getPrompt());
                    return ResponseEntity.ok(response);
                }
            }

            @Service
            public class ModelService {
                private static final String PYTHON_API_URL = "http://localhost:5000/generate";
                
                public String getGeneratedText(String prompt) {
                    RestTemplate restTemplate = new RestTemplate();
                    Map requestBody = new HashMap<>();
                    requestBody.put("prompt", prompt);
                    
                    return restTemplate.postForObject(PYTHON_API_URL, requestBody, String.class);
                }
            }
          
        

This layer handles authentication, logging, and business logic.

Step 8: Create a React Frontend

Build a user-friendly interface for interacting with your model.

          
            
              // Example: React Component
              import React, { useState } from "react";

              function App() {
                const [prompt, setPrompt] = useState("");
                const [response, setResponse] = useState("");
                const [loading, setLoading] = useState(false);

                const handleSubmit = async (e) => {
                  e.preventDefault();
                  setLoading(true);

                  try {
                    const res = await fetch("http://localhost:8080/api/v1/model/generate", {
                      method: "POST",
                      headers: { "Content-Type": "application/json" },
                      body: JSON.stringify({ prompt })
                    });

                    const data = await res.text();
                    setResponse(data);
                  } catch (error) {
                    console.error("Error:", error);
                    setResponse("Something went wrong.");
                  } finally {
                    setLoading(false);
                  }
                };

                return (
                  <div style=<?php echo e(maxWidth: "600px", margin: "50px auto"); ?>>
                    <h1>Domain-Specific LLM</h1>

                    <form onSubmit={handleSubmit}>
                      <textarea
                        rows="4"
                        style=<?php echo e(width: "100%", padding: "10px"); ?>

                        value={prompt}
                        onChange={(e) => setPrompt(e.target.value)}
                      />
                      <button type="submit">
                        {loading ? "Generating..." : "Generate"}
                      </button>
                    </form>

                    {response && (
                      <div>
                        <p>{response}</p>
                      </div>
                    )}
                  </div>
                );
              }

              export default App;
              
          
        

Step 9: Deploy Your Application

Package everything using Docker for consistent deployment.

Production Deployment Considerations:
- GPU inference containers
- Autoscaling (Kubernetes / ECS)
- Model versioning
- Secrets management (Vault / AWS Secrets Manager)
- HTTPS + Auth

          
            # Python backend Dockerfile
            FROM python:3.9
            WORKDIR /app
            COPY requirements.txt .
            RUN pip install -r requirements.txt
            COPY . .
            CMD ["python", "inference_server.py"]
          
        

Use Docker Compose to orchestrate all services, or deploy to cloud platforms like AWS ECS or Google Cloud Run.

Step 10: Monitor and Iterate

After deployment, monitor performance, gather user feedback, and retrain the model as the domain evolves. Optimize system resources, reduce latency, and continuously refine responses to keep your LLM accurate and reliable.

Testing your model:

          
            # Test with various prompts
            test_prompts = [
                "What is the standard treatment for...",
                "Explain the legal implications of...",
                "What are the market trends in..."
            ]

            for prompt in test_prompts:
                response = generate_text({"prompt": prompt})
                print(f"Prompt: {prompt}\nResponse: {response}\n")
          
        

Building a domain-specific LLM involves careful planning, quality data preparation, proper training, and robust deployment infrastructure. By following these steps, you create an AI system that truly understands your field's unique language and requirements.