Exploring the effects of silent data corruption in distributed deep learning training

 

Guardado en:
Detalles Bibliográficos
Autores: Rojas, Elvis, Pérez, Diego, Meneses, Esteban
Formato: artículo preliminar
Fecha de Publicación:2022
Descripción:The profound impact of recent developments in artificial intelligence is unquestionable. The applications of deep learning models are everywhere, from advanced natural language processing to highly accurate prediction of extreme weather. Those models have been continuously increasing in complexity, becoming much more powerful than their original versions. In addition, data to train the models is becoming more available as technological infrastructures sense and collect more readings. Consequently, distributed deep learning training is often times necessary to handle intricate models and massive datasets. Running a distributed training strategy on a supercomputer exposes the models to all the considerations of a large-scale machine; reliability is one of them. As supercomputers integrate a colossal number of components, each fabricated on an ever decreasing feature-size, faults are common during execution of programs. A particular type of fault, silent data corruption, is troublesome because the system does not crash and does not immediately give an evident sign of an error. We set out to explore the effects of that type of faults by inspecting how distributed deep learning training strategies cope with bit-flips that affect their internal data structures. We used checkpoint alteration, a technique that permits the study of this phenomenon on different distributed training platforms and with different deep learning frameworks. We evaluated two distributed learning libraries (Distributed Data Parallel and Horovod) and found out Horovod is slightly more resilient to SDCs. However, fault propagation is similar in both cases, and the model is more sensitive to SDCs than the optimizer.
País:Repositorio UNA
Institución:Universidad Nacional de Costa Rica
Repositorio:Repositorio UNA
Lenguaje:Inglés
OAI Identifier:oai:null:11056/24355
Acceso en línea:http://hdl.handle.net/11056/24355
Palabra clave:PROCESAMIENTO ELECTRÓNICO DE DATOS
SEGURIDAD (INFORMÁTICA)
INTELIGENCIA ARTIFICIAL
APRENDIZAJE PROFUNDO
APRENDIZAJE AUTOMÁTICO
ALGORITMOS
PROCESAMIENTO DE INFORMACIÓN
ENTRENAMIENTO CON DATA PARALLEL
ENTRENAMIENTO CON HOROVOD
ELECTRONIC DATA PROCESSING
SECURITY (COMPUTER SCIENCE)
ARTIFICIAL INTELLIGENCE
DEEP LEARNING
MACHINE LEARNING
ALGORITHMS
INFORMATION PROCESSING
DATA PARALLEL TRAINING
TRAINING WITH HOROVOD