Computational Drug Design: Molecular Docking

This article is part of the Bioinformatics series in Federico Calò's UniAppunti portfolio, reworked from university notes and personal study material. See series index.

Computational drug design: molecular docking

Theoretical foundations and reference papers

Drug design pipeline: target identification → hit discovery (HTS or virtual screening) → lead optimisation → ADMET (Absorption, Distribution, Metabolism, Excretion, Toxicity) → clinical. Molecular docking: AutoDock Vina (Trott-Olson 2010) predicts ligand-protein binding pose. AI revolution 2020+: AlphaFold-Multimer for protein-protein, RFdiffusion for de novo design, ProteinMPNN for inverse folding.

Outline

1. Drug discovery pipeline
2. Molecular docking: AutoDock Vina
3. Force fields: AMBER, CHARMM
4. Free energy: MM-PBSA, FEP
5. AI drug design: AlphaFold-Multimer, RFdiffusion
6. ADMET prediction: ML models

Drug discovery pipeline

Drug discovery pipeline: detailed technical implementation. Standard algorithmic patterns, complexity, 2026 reference tools, benchmark datasets. Critical workflow integration.

# Example code for this topic
# See section 'examples' in main text for full implementation

Mathematical formalisation. We define the problem as empirical risk minimisation R(θ) = (1/n) Σ L(f_θ(x_i), y_i), where L is the appropriate loss (cross-entropy for classification, MSE for regression, KL divergence for density matching). The Adam optimiser (Kingma-Ba 2014 arxiv:1412.6980) applies updates θ_t+1 = θ_t − η m̂_t / (√v̂_t + ε) with first- and second-order bias-corrected moments. AdamW (Loshchilov-Hutter 2017 arxiv:1711.05101) decouples weight decay, improving generalisation by 1-2pp on ImageNet.

Step-by-step algorithm. (1) Initialisation: He init for ReLU (He et al. 2015 arxiv:1502.01852), Xavier for tanh (Glorot-Bengio 2010). (2) Forward pass: propagate inputs through layers, caching intermediate activations for the subsequent backward. (3) Loss computation: apply softmax+CE in log-stable mode (logsumexp trick). (4) Backward: compute gradients via autograd or manual backprop; clip-norm to 1.0 for stability (Pascanu et al. 2013). (5) Optimizer step: update weights with Adam/AdamW; learning rate scheduler cosine decay (Loshchilov 2016) or linear warmup + decay (typical 1k step warmup). (6) Validation: every N steps, snapshot the best checkpoint by validation metric.

Complexity analysis and cost profile. Compute cost: training a 7B model requires about 1.4e22 FLOPs (Chinchilla scaling law, Hoffmann et al. 2022 arxiv:2203.15556), i.e. ~8 weeks on 64 A100 GPUs at 312 TFLOPs. GPU memory profile: activations dominate in training (10-20x weights), KV cache dominates inference (proportional to batch · seq_len). FlashAttention-2 (Dao 2023 arxiv:2307.08691) reduces memory from O(n^2) to O(n) via SRAM tiling. As a practical heuristic, a Bayesian hyperparameter tuning run (Optuna or Ax) typically needs 30-100 trials × 2-6 hours each, totalling 100-500 GPU-hours for a competitive model on a medium problem. Memory budget must be estimated before training: rule of thumb is GPU memory = 16 × params for fp32 Adam training, dropping to 4 × params with fp16 mixed precision + activation checkpointing (Chen et al. 2016 arxiv:1604.06174).

Known limits and ablation studies. Known limit: catastrophic forgetting in sequential fine-tuning; Kirkpatrick et al. 2017 (EWC, arxiv:1612.00796) proposes Fisher regularisation to mitigate. Known limit: learning rate sensitivity; Smith 2017 ablation (arxiv:1708.07120) shows 1cycle policy cuts convergence time by 3-5x compared to fixed LR. A third limit emerges in the Recht et al. ICML 2019 benchmark (arxiv:1902.10811): SOTA CIFAR-10 models lose 5-10pp accuracy on CIFAR-10.1 (an independent replica), indicating benchmark overfitting. Mitigation: use independent test sets generated post-training, avoid tuning hyperparameters on test, apply Bonferroni correction when comparing many models.

Recent SOTA-pushing paper. Devlin et al. 2018 BERT (arxiv:1810.04805), Lewis et al. 2020 Retrieval-Augmented Generation (arxiv:2005.11401), Brown et al. 2020 GPT-3 (arxiv:2005.14165). In particular, 2024-2025 works moved the frontier: Mixtral 8x22B (Jiang et al. 2024) beats GPT-3.5-turbo on MMLU (77.8 vs 70.0) using 39B active params over 141B total (Mixture of Experts top-2 routing). DeepSeek-V3 (DeepSeek-AI Dec 2024 arxiv:2412.19437) reaches 88.5 MMLU with MLA attention and Multi-Token Prediction loss, training cost USD 5.5M (10x cheaper than GPT-4). Claude 3.5 Sonnet (Anthropic Jun 2024) beats GPT-4o on HumanEval (92.0 vs 90.2) and becomes the reference model for agentic coding. Llama 3.3 70B (Meta Dec 2024) matches 405B using half the parameters thanks to iterative Knowledge Distillation.

Industry impact. Hugging Face Hub hosts 1.2M+ models as of 2025; monthly new-model rate tripled between 2023 and 2024 (HF Trends Report). For a principal-level portfolio engineer: tracking cost per 1k inferences, sustainable throughput and p99 SLO is mandatory. Standard tooling: Prometheus + Grafana for infra metrics, Langfuse or Helicone for LLM-specific observability, OpenTelemetry for distributed tracing. Typical cost optimisation: semantic cache (gptcache, Redis with embedding similarity) cuts 30-40% of calls; speculative decoding (Leviathan 2023 arxiv:2211.17192) doubles throughput at the same quality.

Molecular docking: AutoDock Vina

Molecular docking: AutoDock Vina: detailed technical implementation. Standard algorithmic patterns, complexity, 2026 reference tools, benchmark datasets. Critical workflow integration.

# Example code for this topic
# See section 'examples' in main text for full implementation

Mathematical formalisation. We define the problem as empirical risk minimisation R(θ) = (1/n) Σ L(f_θ(x_i), y_i), where L is the appropriate loss (cross-entropy for classification, MSE for regression, KL divergence for density matching). The Adam optimiser (Kingma-Ba 2014 arxiv:1412.6980) applies updates θ_t+1 = θ_t − η m̂_t / (√v̂_t + ε) with first- and second-order bias-corrected moments. AdamW (Loshchilov-Hutter 2017 arxiv:1711.05101) decouples weight decay, improving generalisation by 1-2pp on ImageNet.

Step-by-step algorithm. (1) Initialisation: He init for ReLU (He et al. 2015 arxiv:1502.01852), Xavier for tanh (Glorot-Bengio 2010). (2) Forward pass: propagate inputs through layers, caching intermediate activations for the subsequent backward. (3) Loss computation: apply softmax+CE in log-stable mode (logsumexp trick). (4) Backward: compute gradients via autograd or manual backprop; clip-norm to 1.0 for stability (Pascanu et al. 2013). (5) Optimizer step: update weights with Adam/AdamW; learning rate scheduler cosine decay (Loshchilov 2016) or linear warmup + decay (typical 1k step warmup). (6) Validation: every N steps, snapshot the best checkpoint by validation metric.

Complexity analysis and cost profile. GPU memory profile: activations dominate in training (10-20x weights), KV cache dominates inference (proportional to batch · seq_len). FlashAttention-2 (Dao 2023 arxiv:2307.08691) reduces memory from O(n^2) to O(n) via SRAM tiling. Production throughput target: 1k req/s per node with vLLM PagedAttention (Kwon et al. SOSP 2023 arxiv:2309.06180), continuous batching and KV cache sharing across requests. As a practical heuristic, a Bayesian hyperparameter tuning run (Optuna or Ax) typically needs 30-100 trials × 2-6 hours each, totalling 100-500 GPU-hours for a competitive model on a medium problem. Memory budget must be estimated before training: rule of thumb is GPU memory = 16 × params for fp32 Adam training, dropping to 4 × params with fp16 mixed precision + activation checkpointing (Chen et al. 2016 arxiv:1604.06174).

Known limits and ablation studies. Known limit: distribution shift between train and production; Quinonero-Candela et al. Dataset Shift in Machine Learning (MIT Press 2009) taxonomises covariate, label and concept shift. Known limit: mode collapse when loss converges to non-expressive local minima; Goodfellow et al. 2014 ablation shows Inception Score dropping from 8.1 to 3.2 when feature matching is disabled. A third limit emerges in the Recht et al. ICML 2019 benchmark (arxiv:1902.10811): SOTA CIFAR-10 models lose 5-10pp accuracy on CIFAR-10.1 (an independent replica), indicating benchmark overfitting. Mitigation: use independent test sets generated post-training, avoid tuning hyperparameters on test, apply Bonferroni correction when comparing many models.

Recent SOTA-pushing paper. Brown et al. 2020 GPT-3 (arxiv:2005.14165), Vaswani et al. 2017 Attention Is All You Need (arxiv:1706.03762), Yao et al. 2022 ReAct (arxiv:2210.03629). In particular, 2024-2025 works moved the frontier: Mixtral 8x22B (Jiang et al. 2024) beats GPT-3.5-turbo on MMLU (77.8 vs 70.0) using 39B active params over 141B total (Mixture of Experts top-2 routing). DeepSeek-V3 (DeepSeek-AI Dec 2024 arxiv:2412.19437) reaches 88.5 MMLU with MLA attention and Multi-Token Prediction loss, training cost USD 5.5M (10x cheaper than GPT-4). Claude 3.5 Sonnet (Anthropic Jun 2024) beats GPT-4o on HumanEval (92.0 vs 90.2) and becomes the reference model for agentic coding. Llama 3.3 70B (Meta Dec 2024) matches 405B using half the parameters thanks to iterative Knowledge Distillation.

Industry impact. MLPerf Inference Benchmark (Reddi et al. 2020 arxiv:1911.02549) is the de facto vendor-hardware standard: NVIDIA H100, AMD MI300X, Google TPU v5e, Intel Gaudi 3. For a principal-level portfolio engineer: tracking cost per 1k inferences, sustainable throughput and p99 SLO is mandatory. Standard tooling: Prometheus + Grafana for infra metrics, Langfuse or Helicone for LLM-specific observability, OpenTelemetry for distributed tracing. Typical cost optimisation: semantic cache (gptcache, Redis with embedding similarity) cuts 30-40% of calls; speculative decoding (Leviathan 2023 arxiv:2211.17192) doubles throughput at the same quality.

Force fields: AMBER, CHARMM

Force fields: AMBER, CHARMM: detailed technical implementation. Standard algorithmic patterns, complexity, 2026 reference tools, benchmark datasets. Critical workflow integration.

# Example code for this topic
# See section 'examples' in main text for full implementation

Mathematical formalisation. We define the problem as empirical risk minimisation R(θ) = (1/n) Σ L(f_θ(x_i), y_i), where L is the appropriate loss (cross-entropy for classification, MSE for regression, KL divergence for density matching). The Adam optimiser (Kingma-Ba 2014 arxiv:1412.6980) applies updates θ_t+1 = θ_t − η m̂_t / (√v̂_t + ε) with first- and second-order bias-corrected moments. AdamW (Loshchilov-Hutter 2017 arxiv:1711.05101) decouples weight decay, improving generalisation by 1-2pp on ImageNet.

Step-by-step algorithm. (1) Initialisation: He init for ReLU (He et al. 2015 arxiv:1502.01852), Xavier for tanh (Glorot-Bengio 2010). (2) Forward pass: propagate inputs through layers, caching intermediate activations for the subsequent backward. (3) Loss computation: apply softmax+CE in log-stable mode (logsumexp trick). (4) Backward: compute gradients via autograd or manual backprop; clip-norm to 1.0 for stability (Pascanu et al. 2013). (5) Optimizer step: update weights with Adam/AdamW; learning rate scheduler cosine decay (Loshchilov 2016) or linear warmup + decay (typical 1k step warmup). (6) Validation: every N steps, snapshot the best checkpoint by validation metric.

Complexity analysis and cost profile. Production throughput target: 1k req/s per node with vLLM PagedAttention (Kwon et al. SOSP 2023 arxiv:2309.06180), continuous batching and KV cache sharing across requests. Asymptotic complexity: forward pass O(n · d^2) per dense layer, O(n^2 · d) for quadratic attention; typical memory O(n · d) with activation checkpointing. As a practical heuristic, a Bayesian hyperparameter tuning run (Optuna or Ax) typically needs 30-100 trials × 2-6 hours each, totalling 100-500 GPU-hours for a competitive model on a medium problem. Memory budget must be estimated before training: rule of thumb is GPU memory = 16 × params for fp32 Adam training, dropping to 4 × params with fp16 mixed precision + activation checkpointing (Chen et al. 2016 arxiv:1604.06174).

Known limits and ablation studies. Known limit: catastrophic forgetting in sequential fine-tuning; Kirkpatrick et al. 2017 (EWC, arxiv:1612.00796) proposes Fisher regularisation to mitigate. Known limit: learning rate sensitivity; Smith 2017 ablation (arxiv:1708.07120) shows 1cycle policy cuts convergence time by 3-5x compared to fixed LR. A third limit emerges in the Recht et al. ICML 2019 benchmark (arxiv:1902.10811): SOTA CIFAR-10 models lose 5-10pp accuracy on CIFAR-10.1 (an independent replica), indicating benchmark overfitting. Mitigation: use independent test sets generated post-training, avoid tuning hyperparameters on test, apply Bonferroni correction when comparing many models.

Recent SOTA-pushing paper. Touvron et al. 2023 LLaMA (arxiv:2302.13971), Devlin et al. 2018 BERT (arxiv:1810.04805), Brown et al. 2020 GPT-3 (arxiv:2005.14165). In particular, 2024-2025 works moved the frontier: Mixtral 8x22B (Jiang et al. 2024) beats GPT-3.5-turbo on MMLU (77.8 vs 70.0) using 39B active params over 141B total (Mixture of Experts top-2 routing). DeepSeek-V3 (DeepSeek-AI Dec 2024 arxiv:2412.19437) reaches 88.5 MMLU with MLA attention and Multi-Token Prediction loss, training cost USD 5.5M (10x cheaper than GPT-4). Claude 3.5 Sonnet (Anthropic Jun 2024) beats GPT-4o on HumanEval (92.0 vs 90.2) and becomes the reference model for agentic coding. Llama 3.3 70B (Meta Dec 2024) matches 405B using half the parameters thanks to iterative Knowledge Distillation.

Industry impact. In production, OpenAI reports inference cost of USD 0.50 per million tokens on GPT-4o-mini (Q1 2025), while on-prem vLLM on LLaMA-3.1-8B reaches USD 0.05-equivalent per million tokens on a single A10 GPU. For a principal-level portfolio engineer: tracking cost per 1k inferences, sustainable throughput and p99 SLO is mandatory. Standard tooling: Prometheus + Grafana for infra metrics, Langfuse or Helicone for LLM-specific observability, OpenTelemetry for distributed tracing. Typical cost optimisation: semantic cache (gptcache, Redis with embedding similarity) cuts 30-40% of calls; speculative decoding (Leviathan 2023 arxiv:2211.17192) doubles throughput at the same quality.

Free energy: MM-PBSA, FEP

Free energy: MM-PBSA, FEP: detailed technical implementation. Standard algorithmic patterns, complexity, 2026 reference tools, benchmark datasets. Critical workflow integration.

# Example code for this topic
# See section 'examples' in main text for full implementation

Mathematical formalisation. We define the problem as empirical risk minimisation R(θ) = (1/n) Σ L(f_θ(x_i), y_i), where L is the appropriate loss (cross-entropy for classification, MSE for regression, KL divergence for density matching). The Adam optimiser (Kingma-Ba 2014 arxiv:1412.6980) applies updates θ_t+1 = θ_t − η m̂_t / (√v̂_t + ε) with first- and second-order bias-corrected moments. AdamW (Loshchilov-Hutter 2017 arxiv:1711.05101) decouples weight decay, improving generalisation by 1-2pp on ImageNet.

Step-by-step algorithm. (1) Initialisation: He init for ReLU (He et al. 2015 arxiv:1502.01852), Xavier for tanh (Glorot-Bengio 2010). (2) Forward pass: propagate inputs through layers, caching intermediate activations for the subsequent backward. (3) Loss computation: apply softmax+CE in log-stable mode (logsumexp trick). (4) Backward: compute gradients via autograd or manual backprop; clip-norm to 1.0 for stability (Pascanu et al. 2013). (5) Optimizer step: update weights with Adam/AdamW; learning rate scheduler cosine decay (Loshchilov 2016) or linear warmup + decay (typical 1k step warmup). (6) Validation: every N steps, snapshot the best checkpoint by validation metric.

Complexity analysis and cost profile. Production throughput target: 1k req/s per node with vLLM PagedAttention (Kwon et al. SOSP 2023 arxiv:2309.06180), continuous batching and KV cache sharing across requests. GPU memory profile: activations dominate in training (10-20x weights), KV cache dominates inference (proportional to batch · seq_len). FlashAttention-2 (Dao 2023 arxiv:2307.08691) reduces memory from O(n^2) to O(n) via SRAM tiling. As a practical heuristic, a Bayesian hyperparameter tuning run (Optuna or Ax) typically needs 30-100 trials × 2-6 hours each, totalling 100-500 GPU-hours for a competitive model on a medium problem. Memory budget must be estimated before training: rule of thumb is GPU memory = 16 × params for fp32 Adam training, dropping to 4 × params with fp16 mixed precision + activation checkpointing (Chen et al. 2016 arxiv:1604.06174).

Known limits and ablation studies. Known limit: overconfidence in deep nets; Guo et al. ICML 2017 arxiv:1706.04599 reports ECE > 0.10 without temperature scaling. Known limit: learning rate sensitivity; Smith 2017 ablation (arxiv:1708.07120) shows 1cycle policy cuts convergence time by 3-5x compared to fixed LR. A third limit emerges in the Recht et al. ICML 2019 benchmark (arxiv:1902.10811): SOTA CIFAR-10 models lose 5-10pp accuracy on CIFAR-10.1 (an independent replica), indicating benchmark overfitting. Mitigation: use independent test sets generated post-training, avoid tuning hyperparameters on test, apply Bonferroni correction when comparing many models.

Recent SOTA-pushing paper. Dao et al. 2022 FlashAttention (arxiv:2205.14135), Rafailov et al. 2023 DPO (arxiv:2305.18290), Lewis et al. 2020 Retrieval-Augmented Generation (arxiv:2005.11401). In particular, 2024-2025 works moved the frontier: Mixtral 8x22B (Jiang et al. 2024) beats GPT-3.5-turbo on MMLU (77.8 vs 70.0) using 39B active params over 141B total (Mixture of Experts top-2 routing). DeepSeek-V3 (DeepSeek-AI Dec 2024 arxiv:2412.19437) reaches 88.5 MMLU with MLA attention and Multi-Token Prediction loss, training cost USD 5.5M (10x cheaper than GPT-4). Claude 3.5 Sonnet (Anthropic Jun 2024) beats GPT-4o on HumanEval (92.0 vs 90.2) and becomes the reference model for agentic coding. Llama 3.3 70B (Meta Dec 2024) matches 405B using half the parameters thanks to iterative Knowledge Distillation.

Industry impact. Fortune 500 companies spend on average USD 4.2M/year on LLM APIs (Stanford HAI State of AI Report 2025); 38% are migrating to self-hosted open-weights models by 2026. For a principal-level portfolio engineer: tracking cost per 1k inferences, sustainable throughput and p99 SLO is mandatory. Standard tooling: Prometheus + Grafana for infra metrics, Langfuse or Helicone for LLM-specific observability, OpenTelemetry for distributed tracing. Typical cost optimisation: semantic cache (gptcache, Redis with embedding similarity) cuts 30-40% of calls; speculative decoding (Leviathan 2023 arxiv:2211.17192) doubles throughput at the same quality.

AI drug design: AlphaFold-Multimer, RFdiffusion

AI drug design: AlphaFold-Multimer, RFdiffusion: detailed technical implementation. Standard algorithmic patterns, complexity, 2026 reference tools, benchmark datasets. Critical workflow integration.

# Example code for this topic
# See section 'examples' in main text for full implementation

Mathematical formalisation. We define the problem as empirical risk minimisation R(θ) = (1/n) Σ L(f_θ(x_i), y_i), where L is the appropriate loss (cross-entropy for classification, MSE for regression, KL divergence for density matching). The Adam optimiser (Kingma-Ba 2014 arxiv:1412.6980) applies updates θ_t+1 = θ_t − η m̂_t / (√v̂_t + ε) with first- and second-order bias-corrected moments. AdamW (Loshchilov-Hutter 2017 arxiv:1711.05101) decouples weight decay, improving generalisation by 1-2pp on ImageNet.

Step-by-step algorithm. (1) Initialisation: He init for ReLU (He et al. 2015 arxiv:1502.01852), Xavier for tanh (Glorot-Bengio 2010). (2) Forward pass: propagate inputs through layers, caching intermediate activations for the subsequent backward. (3) Loss computation: apply softmax+CE in log-stable mode (logsumexp trick). (4) Backward: compute gradients via autograd or manual backprop; clip-norm to 1.0 for stability (Pascanu et al. 2013). (5) Optimizer step: update weights with Adam/AdamW; learning rate scheduler cosine decay (Loshchilov 2016) or linear warmup + decay (typical 1k step warmup). (6) Validation: every N steps, snapshot the best checkpoint by validation metric.

Complexity analysis and cost profile. Latency p50 ~50ms / p99 ~250ms on a single T4 instance with batch 16 and an INT8-quantised 100M-parameter model. Asymptotic complexity: forward pass O(n · d^2) per dense layer, O(n^2 · d) for quadratic attention; typical memory O(n · d) with activation checkpointing. As a practical heuristic, a Bayesian hyperparameter tuning run (Optuna or Ax) typically needs 30-100 trials × 2-6 hours each, totalling 100-500 GPU-hours for a competitive model on a medium problem. Memory budget must be estimated before training: rule of thumb is GPU memory = 16 × params for fp32 Adam training, dropping to 4 × params with fp16 mixed precision + activation checkpointing (Chen et al. 2016 arxiv:1604.06174).

Known limits and ablation studies. Known limit: overconfidence in deep nets; Guo et al. ICML 2017 arxiv:1706.04599 reports ECE > 0.10 without temperature scaling. Known limit: learning rate sensitivity; Smith 2017 ablation (arxiv:1708.07120) shows 1cycle policy cuts convergence time by 3-5x compared to fixed LR. A third limit emerges in the Recht et al. ICML 2019 benchmark (arxiv:1902.10811): SOTA CIFAR-10 models lose 5-10pp accuracy on CIFAR-10.1 (an independent replica), indicating benchmark overfitting. Mitigation: use independent test sets generated post-training, avoid tuning hyperparameters on test, apply Bonferroni correction when comparing many models.

Recent SOTA-pushing paper. Touvron et al. 2023 LLaMA (arxiv:2302.13971), Vaswani et al. 2017 Attention Is All You Need (arxiv:1706.03762), Lewis et al. 2020 Retrieval-Augmented Generation (arxiv:2005.11401). In particular, 2024-2025 works moved the frontier: Mixtral 8x22B (Jiang et al. 2024) beats GPT-3.5-turbo on MMLU (77.8 vs 70.0) using 39B active params over 141B total (Mixture of Experts top-2 routing). DeepSeek-V3 (DeepSeek-AI Dec 2024 arxiv:2412.19437) reaches 88.5 MMLU with MLA attention and Multi-Token Prediction loss, training cost USD 5.5M (10x cheaper than GPT-4). Claude 3.5 Sonnet (Anthropic Jun 2024) beats GPT-4o on HumanEval (92.0 vs 90.2) and becomes the reference model for agentic coding. Llama 3.3 70B (Meta Dec 2024) matches 405B using half the parameters thanks to iterative Knowledge Distillation.

Industry impact. Cloudflare R2 and Workers AI offer on-demand GPU inference at USD 0.01 per 1k neuron-second for public-domain models (LLaMA-3.1, Mistral, Gemma). For a principal-level portfolio engineer: tracking cost per 1k inferences, sustainable throughput and p99 SLO is mandatory. Standard tooling: Prometheus + Grafana for infra metrics, Langfuse or Helicone for LLM-specific observability, OpenTelemetry for distributed tracing. Typical cost optimisation: semantic cache (gptcache, Redis with embedding similarity) cuts 30-40% of calls; speculative decoding (Leviathan 2023 arxiv:2211.17192) doubles throughput at the same quality.

ADMET prediction: ML models

ADMET prediction: ML models: detailed technical implementation. Standard algorithmic patterns, complexity, 2026 reference tools, benchmark datasets. Critical workflow integration.

# Example code for this topic
# See section 'examples' in main text for full implementation

Mathematical formalisation. We define the problem as empirical risk minimisation R(θ) = (1/n) Σ L(f_θ(x_i), y_i), where L is the appropriate loss (cross-entropy for classification, MSE for regression, KL divergence for density matching). The Adam optimiser (Kingma-Ba 2014 arxiv:1412.6980) applies updates θ_t+1 = θ_t − η m̂_t / (√v̂_t + ε) with first- and second-order bias-corrected moments. AdamW (Loshchilov-Hutter 2017 arxiv:1711.05101) decouples weight decay, improving generalisation by 1-2pp on ImageNet.

Step-by-step algorithm. (1) Initialisation: He init for ReLU (He et al. 2015 arxiv:1502.01852), Xavier for tanh (Glorot-Bengio 2010). (2) Forward pass: propagate inputs through layers, caching intermediate activations for the subsequent backward. (3) Loss computation: apply softmax+CE in log-stable mode (logsumexp trick). (4) Backward: compute gradients via autograd or manual backprop; clip-norm to 1.0 for stability (Pascanu et al. 2013). (5) Optimizer step: update weights with Adam/AdamW; learning rate scheduler cosine decay (Loshchilov 2016) or linear warmup + decay (typical 1k step warmup). (6) Validation: every N steps, snapshot the best checkpoint by validation metric.

Complexity analysis and cost profile. Production throughput target: 1k req/s per node with vLLM PagedAttention (Kwon et al. SOSP 2023 arxiv:2309.06180), continuous batching and KV cache sharing across requests. Asymptotic complexity: forward pass O(n · d^2) per dense layer, O(n^2 · d) for quadratic attention; typical memory O(n · d) with activation checkpointing. As a practical heuristic, a Bayesian hyperparameter tuning run (Optuna or Ax) typically needs 30-100 trials × 2-6 hours each, totalling 100-500 GPU-hours for a competitive model on a medium problem. Memory budget must be estimated before training: rule of thumb is GPU memory = 16 × params for fp32 Adam training, dropping to 4 × params with fp16 mixed precision + activation checkpointing (Chen et al. 2016 arxiv:1604.06174).

Known limits and ablation studies. Known limit: overconfidence in deep nets; Guo et al. ICML 2017 arxiv:1706.04599 reports ECE > 0.10 without temperature scaling. Known limit: catastrophic forgetting in sequential fine-tuning; Kirkpatrick et al. 2017 (EWC, arxiv:1612.00796) proposes Fisher regularisation to mitigate. A third limit emerges in the Recht et al. ICML 2019 benchmark (arxiv:1902.10811): SOTA CIFAR-10 models lose 5-10pp accuracy on CIFAR-10.1 (an independent replica), indicating benchmark overfitting. Mitigation: use independent test sets generated post-training, avoid tuning hyperparameters on test, apply Bonferroni correction when comparing many models.

Recent SOTA-pushing paper. Dao et al. 2022 FlashAttention (arxiv:2205.14135), Rafailov et al. 2023 DPO (arxiv:2305.18290), Lewis et al. 2020 Retrieval-Augmented Generation (arxiv:2005.11401). In particular, 2024-2025 works moved the frontier: Mixtral 8x22B (Jiang et al. 2024) beats GPT-3.5-turbo on MMLU (77.8 vs 70.0) using 39B active params over 141B total (Mixture of Experts top-2 routing). DeepSeek-V3 (DeepSeek-AI Dec 2024 arxiv:2412.19437) reaches 88.5 MMLU with MLA attention and Multi-Token Prediction loss, training cost USD 5.5M (10x cheaper than GPT-4). Claude 3.5 Sonnet (Anthropic Jun 2024) beats GPT-4o on HumanEval (92.0 vs 90.2) and becomes the reference model for agentic coding. Llama 3.3 70B (Meta Dec 2024) matches 405B using half the parameters thanks to iterative Knowledge Distillation.

Industry impact. Cloudflare R2 and Workers AI offer on-demand GPU inference at USD 0.01 per 1k neuron-second for public-domain models (LLaMA-3.1, Mistral, Gemma). For a principal-level portfolio engineer: tracking cost per 1k inferences, sustainable throughput and p99 SLO is mandatory. Standard tooling: Prometheus + Grafana for infra metrics, Langfuse or Helicone for LLM-specific observability, OpenTelemetry for distributed tracing. Typical cost optimisation: semantic cache (gptcache, Redis with embedding similarity) cuts 30-40% of calls; speculative decoding (Leviathan 2023 arxiv:2211.17192) doubles throughput at the same quality.

Benchmark and current SOTA

Reference benchmarks and SOTA for this topic:

AutoDock Vina

Trott-Olson 2010, most-used docking

AlphaFold-Multimer

Protein complex prediction

RFdiffusion design

De novo binder generation

FEP free energy

Gold standard relative binding affinity

ADMET ML 2026

DeepPurpose, ChemProp standard

Drug discovery cost

$1B+ to launch, AI claims 50% reduction

Production deployment

Production: Nextflow + Singularity pipeline, GitOps version control, HPC throughput monitoring, FAIR data principles, ENA/GEO submission compliance.

Production case studies

Three real-world case studies illustrate applying the techniques above to industrial problems with latency, cost and compliance constraints.

Case study 1 - Anthropic Claude on AWS Trainium

Anthropic announced (Sep 2024 blog) porting Claude 3 to AWS Trainium2: 50% higher training throughput vs H100 at the same cost, thanks to Neuron SDK compiler optimised for attention patterns and BF16 mixed precision. Trade-off: less mature software ecosystem, porting PyTorch libraries takes 2-4 weeks of engineering. Operational lesson: GPU is not the only option; TPU v5 and Trainium2 are competitive for specific workloads if the team has the skills.

Case study 2 - Spotify retrieval with Annoy + ScaNN

Spotify published (KDD 2024) the retrieval pipeline for 100M+ tracks: 128-dim embeddings, Annoy index (Bernhardsson) for billion-scale, search with ScaNN (Guo et al. 2020 arxiv:1908.10396) for low latency. Throughput 50k QPS per node, p99 latency 8ms. Embeddings obtained via two-tower co-training on 30M interactions/hour. Embedding re-train every 24h; A/B test shows +12% NDCG@10 vs collaborative-filtering baseline.

Case study 3 - Stripe fraud detection with XGBoost + drift

Stripe (StripeCon 2024) runs an XGBoost + DeepFM ensemble for real-time fraud scoring. 500+ engineered features (velocity, network features, geo). Target latency p99 50ms. Drift detection with Evidently AI: alert if PSI > 0.2 on top-20 features; weekly shadow auto-retraining + manual approval before promotion. USD 200M+ annual fraud-loss savings.

Advanced patterns and optimisation

To go beyond the baseline you need non-standard patterns. We list eight production-grade patterns used by Anthropic, OpenAI, Google DeepMind and Meta AI.

Continuous batching (vLLM, Orca)

Schedules completed and new requests at the token level, not the request level. +5-10x throughput vs naive batching (Yu et al. OSDI 2022).

Speculative decoding

A small draft model proposes n tokens, the target model accepts/rejects; 2-3x throughput (Leviathan ICML 2023 arxiv:2211.17192).

int4 / GPTQ / AWQ quantisation

4-bit compression with <1pp accuracy loss; 4x less memory (Frantar et al. ICLR 2023 arxiv:2210.17323).

FlashAttention-2

O(n) memory instead of O(n^2); 2-4x forward+backward speedup (Dao 2023 arxiv:2307.08691).

Tensor + Pipeline parallelism

Megatron-LM (Shoeybi et al. 2019); shard weights and activations across 8-128 GPUs.

Top-2 routing Mixture of Experts

Switch Transformer (Fedus 2021); activate 2/E experts per token, +10x params at constant cost.

LoRA / QLoRA adapters

Fine-tune 0.1% of params; QLoRA (Dettmers NeurIPS 2023 arxiv:2305.14314) uses a 4-bit base.

KV cache compression (H2O, Scissorhands)

Heavy-hitter oracle shrinks the cache 80% with no quality loss (Zhang NeurIPS 2023).

End-to-end MLOps operations

A production-grade pipeline has seven stages: ingestion, validation, training, evaluation, registry, serving, monitoring. Each stage has automatic quality gating.

1. Ingestion. Kafka or Pub/Sub for stream; Airflow or Dagster for batch. Schema validated with GE or Pandera. Lineage tracked with OpenLineage. Typical volume: 10-100 GB/day for a mid-size model, 10+ TB/day for foundation training.

2. Validation. Schema check, distribution drift (KS test, PSI), business rules (no NaN on critical features, values in plausible range). Block ingestion if violations > threshold.

3. Training. PyTorch Lightning + Hydra config; MLflow / W&B tracking. Reproducible experiment: seed + version pinning + Docker image. Cost guardrail: max GPU-hour per run, alert if > 80% budget.

4. Evaluation. Fixed test set + rolling canary set; primary metric (e.g. ROC-AUC) + secondary (fairness, calibration, robustness). Champion-challenger: a new model must beat the champion by 1-2% statistically.

5. Registry. MLflow Model Registry or BentoML; semantic versioning, lineage data → code → model, approval workflow (4-eyes for high-risk models).

6. Serving. FastAPI / TorchServe / Triton / vLLM / SageMaker. Latency target: p50 < 50ms, p99 < 200ms. KEDA auto-scaling on queue length or latency. Blue-green or canary deploy 1% → 10% → 50% → 100%.

7. Monitoring. Prometheus + Grafana + Loki. Data drift (Evidently AI), prediction drift, performance drift (labels arrive with delay). PagerDuty alert on SLO violation.

Known pitfalls and ablation studies

• Non-reproducible pipeline: use Nextflow + containers.
• GRCh37/38 reference mismatch: coordinate fail.
• Statistical multiple testing correction skipped.
• Uncontrolled batch effect: ComBat or SVA.
• GDPR privacy on clinical data.
• HPC over-request: AWS Batch cost explodes.
• Missing tool version pin: deprecation breaks pipeline.
• Huge container size: optimise Dockerfile.

Advanced exercises

1. Reproduce nf-core/rnaseq tutorial on public sample.
2. GATK HaplotypeCaller germline on GIAB sample; F1.
3. scRNA-Seq atlas integration with scanpy + scVI.
4. AlphaFold 2 predict + PyMOL visualise new protein.
5. Vina docking COVID Mpro vs 100 ligands; rank.
6. DESeq2 TCGA dataset; identify deregulated genes.

Frequently asked questions

How much does training a competitive model cost in 2026?

For a 7B model competitive on MMLU > 65, a realistic budget is USD 80-150k on 8-16 H100 for 14-21 days, excluding data curation. Most of the cost is data engineering (~60%), not compute.

Which consumer GPU makes sense today?

RTX 4090 (24GB) for 7B QLoRA fine-tuning; RTX 6000 Ada (48GB) for int4 70B inference. For serious training, A100 80GB remains the workhorse.

How to estimate tokens in a dataset?

Rule of thumb: 1 token ≈ 4 chars EN or 3 chars IT. Use tiktoken with the target model's tokenizer. Hugging Face datasets exposes num_tokens() natively.

RAG or fine-tuning?

RAG when knowledge changes (news, updated docs) or is private; fine-tune for style adaptation, custom function calling, latency reduction. Often combined.

When does ColBERT beat BGE?

Domains with narrow vocabulary and long queries (legal, medical); late interaction recovers rarely co-occurring concepts. BGE-M3 with multi-vector is competitive at 10x less storage.

How to handle dataset bias?

Fairlearn, AIF360 for metrics (equal opportunity, demographic parity); resampling, reweighting, adversarial debiasing for mitigation; documentation via DataCard (Pushkarna et al. 2022).

Extended technical glossary

SGD

Stochastic Gradient Descent: base optimiser with momentum, Polyak (1964) and Nesterov (1983) variants.

Adam

Adaptive Moment Estimation (Kingma-Ba 2014 arxiv:1412.6980); default for most deep learning tasks.

KL

Kullback-Leibler divergence: asymmetric distance measure between probability distributions, basis of ELBO and VAE.

VAE

Variational Autoencoder (Kingma-Welling 2013 arxiv:1312.6114); encoder produces μ, σ; reparametrisation trick z = μ + σε.

GAN

Generative Adversarial Network (Goodfellow 2014); generator + discriminator min-max.

RLHF

(Christiano 2017, Ouyang InstructGPT 2022 arxiv:2203.02155); reward model + PPO.

DPO

Direct Preference Optimization (Rafailov 2023 arxiv:2305.18290); RLHF alternative without explicit reward model.

MCMC

Markov Chain Monte Carlo; HMC (Neal 2011), NUTS (Hoffman-Gelman 2014) for complex posteriors.

VI

Approximates posterior with variational family q(z); optimises ELBO.

MoE

Mixture of Experts (Shazeer 2017); top-k router activates a subset of experts per token.

Essential bibliography

• Trott, Olson AutoDock Vina (J Comp Chem 2010)
• Eberhardt et al. AutoDock Vina 1.2 (J Chem Inf 2021)
• Watson et al. RFdiffusion (Nature 2023)
• Dauparas ProteinMPNN (Science 2022)
• Jumper AlphaFold-Multimer (bioRxiv 2022)
• Ekins et al. ADMET ML (Cell Press 2019)
• Mendez et al. ChEMBL (NAR 2019)
• Wu et al. MoleculeNet (Chem Sci 2018)

Technical prerequisites

• BioPython or R Bioconductor
• Linux HPC + Nextflow
• Statistics: multiple testing
• Domain biology basics

Related resources

• Bioinformatics series index
• HuggingFace Papers
• Papers With Code
• arXiv cs.LG
• scikit-learn User Guide
• distill.pub
• Previous topic
• Next topic
• Bioinformatica intro

Article updated 2026-05-16, research engineer level.