Frontiers

OpenMP-annotated code dataset for large language model fine-tuning on parallel programming tasks

High-performance computing (HPC) plays a critical role in scientific discovery, engineering simulation, and data-intensive applications. Open Multi-Processing (OpenMP) is one of the most widely ...
InfoQ

Meta Details GEM Ads Model Using LLM-Scale Training, Hybrid Parallelism, and Knowledge Transfer

Unlock the full InfoQ experience by logging in! Stay updated with your favorite authors and topics, engage with content, and download exclusive resources. Dany Lepage discusses the architectural ...
blockchain

NVIDIA NVL72: Revolutionizing MoE Model Scaling with Expert Parallelism

NVIDIA's NVL72 systems are transforming large-scale MoE model deployment by introducing Wide Expert Parallelism, optimizing performance and reducing costs. NVIDIA is advancing the deployment of ...
IEEE

Research on Model Parallelism and Data Parallelism Optimization Methods in Large Language Model—Based Recommendation Systems

Abstract: With the rapid adoption of large language models (LLMs) in recommendation systems, the computational and communication bottlenecks caused by their massive parameter sizes and large data ...
GitHub

Bitsandbytes quantization for litgpt 2d parallel model (TP+FSDP) within LightningTrainer

I'm trying to run inference within the LightningTrainer using a litgpt model with 2d parallelization (TP+FSDP) while using a Bitsandbytes precision plugin to enable quantization, however I get into ...
TechCrunch

Google rolls out Gemini Deep Think AI, a reasoning model that tests multiple ideas in parallel

Google DeepMind is rolling out Gemini 2.5 Deep Think, which, the company says, is its most advanced AI reasoning model, able to answer questions by exploring and considering multiple ideas ...