Towards Diverse Device Heterogeneous Federated Learning via Task Arithmetic Knowledge Integration

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21230%2F24%3A00383683" target="_blank" >RIV/68407700:21230/24:00383683 - isvavai.cz</a>

Result on the web

<a href="https://proceedings.neurips.cc/paper_files/paper/2024/hash/e6d1d6195f6f3e32a930643e0ef46332-Abstract-Conference.html" target="_blank" >https://proceedings.neurips.cc/paper_files/paper/2024/hash/e6d1d6195f6f3e32a930643e0ef46332-Abstract-Conference.html</a>

DOI - Digital Object Identifier

—

Result language

angličtina

Original language name

Towards Diverse Device Heterogeneous Federated Learning via Task Arithmetic Knowledge Integration

Original language description

Federated Learning (FL) has emerged as a promising paradigm for collaborative machine learning, while preserving user data privacy. Despite its potential, standard FL algorithms lack support for diverse heterogeneous device prototypes, which vary significantly in model and dataset sizes-from small IoT devices to large workstations. This limitation is only partially addressed by existing knowledge distillation (KD) techniques, which often fail to transfer knowledge effectively across a broad spectrum of device prototypes with varied capabilities. This failure primarily stems from two issues: the dilution of informative logits from more capable devices by those from less capable ones, and the use of a single integrated logits as the distillation target across all devices, which neglects their individual learning capacities and and the unique contributions of each device. To address these challenges, we introduce TAKFL, a novel KD-based framework that treats the knowledge transfer from each device prototype's ensemble as a separate task, independently distilling each to preserve its unique contributions and avoid dilution. TAKFL also incorporates a KD-based self-regularization technique to mitigate the issues related to the noisy and unsupervised ensemble distillation process. To integrate the separately distilled knowledge, we introduce an adaptive task arithmetic knowledge integration process, allowing each student model to customize the knowledge integration for optimal performance. Additionally, we present theoretical results demonstrating the effectiveness of task arithmetic in transferring knowledge across heterogeneous device prototypes with varying capacities. Comprehensive evaluations of our method across both computer vision (CV) and natural language processing (NLP) tasks demonstrate that TAKFL achieves state-of-the-art results in a variety of datasets and settings, significantly outperforming existing KD-based methods. Our code is released at https://github.com/MMorafah/TAKFL.

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

Result was created during the realization of more than one project. More information in the Projects tab.

Continuities

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Publication year

2024

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

Advances in Neural Information Processing Systems 37 (NeurIPS 2024)

ISBN

9798331314385

ISSN

1049-5258

e-ISSN

1049-5258

Number of pages

44

Pages from-to

—

Publisher name

Neural Information Processing Systems Foundation, Inc.

Place of publication

—

Event location

Vancouver

Event date

Dec 10, 2024

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

001633253000129