Federated Transfer Learning-Based Paper Breakage Fault Diagnosis

2.1. Data In papermaking , parameters such as paper product basis weight, sizing flow rate, and machine speed vary. This paper analyzes key process parameters of the papermaking machine, classifies different production processes based on quantitative set values, and merges similar operating conditions according to other key process variables to facilitate subsequent modeling analysis. For the papermaking process, the main difference between operating conditions lies in the production process, and different production processes correspond to different process parameters. This study segments the collected dataset based on variations in basis weight set values, resulting in multiple time series data subsets, and analyzes the relationships between basis weight and other variables that strongly influence the production process. After segmenting the operating condition data, the study classifies the conditions based on three parameters: basis weight, machine speed, and long fiber ratio. Seven distinct production conditions are identified by splitting and reorganizing the existing dataset The three conditions with the largest amount of data are designated as Condition A, Condition B, and Condition C, respectively, for data preparation in subsequent modeling. 2.2. Modeling System 2.2.1. Model Building In the paper production process, changes in papermaking machine parameters can significantly impact the production process. To explore the relationship between paper breakage faults and operational parameters and their variations and to quickly locate the fault, actual production data is obtained and processed. Based on fault analysis mechanisms and experience, unsupervised clustering methods are used. These methods categorize faults and identify different types of paper breakage issues. Since process parameters vary across different production conditions, different algorithms are used to build and train fault detection models for each production state. Various evaluation metrics are employed to assess the effectiveness of the models. For high-dimensional data, different clustering methods can be chosen based on the scenario, including partition clustering, density clustering, distribution clustering, and hierarchical clustering [24]. K-Means clustering is simple, easy to implement, and converges quickly, but it is sensitive to outliers, which can affect the clustering results. It aims to partition the dataset into K non-overlapping subsets, minimizing the sum of distances between data points and their corresponding cluster centers [25]. Based on probability density, the Gaussian Mixture Model (GMM)performs better with multidimensional data. Using both K-Means and GMM for clustering allows for cross-validation of results, improving the reliability of the clustering outcome. To determine the optimal number of clusters, this study employs both methods and compares their results for a comprehensive evaluation.In analyzing the papermaking process, it is crucial to initially determine the operating status of the papermaking machine based on paper break signals. Given the diversity of paper break fault types, fault diagnosis requires classifying these various fault types. Commonly employed fault classification models include Logistic Regression (LR) [26], Support Vector Machines (SVM) [27], Random Forest (RF) [28], and Softmax Classifier. Logistic regression fits model parameters through maximum likelihood estimation. Given the training data, optimization algorithms like gradient descent are used. They iteratively adjust the model parameters. This process minimizes the discrepancy between the model’s predicted values and the actual observed values [29]. Support Vector Machines (SVMs) perform exceptionally well in classification tasks. The objective is to find an optimal hyperplane. This hyperplane separates data from different classes. It aims to maximize the margin between the classes. Additionally, it seeks to minimize the number of misclassified points [30]. Random Forest is an ensemble method that enhances model performance and robustness by constructing multiple decision trees [31]. A decision tree is a machine learning model based on a tree structure used for classification and regression analysis. It builds a tree-like decision diagram by recursively and binary splitting the dataset. The construction process involves selecting the optimal features for splitting until a predefined stopping criterion is met. Its advantages include minimal preprocessing requirements, ease of understanding and interpretation, and the ability to handle both numerical and categorical data [32]. An autoencoder is an unsupervised learning model that compresses data into a lower-dimensional representation and reconstructs the input from this compressed form [33]. In the training process of an autoencoder, the objective is to minimize the reconstruction error and derive a lower-dimensional representation of the data, thereby facilitating feature extraction from high-dimensional inputs. For multi-class classification problems, the Softmax classifier is frequently employed. This classifier produces probabilistic outputs by applying the Softmax function, which exponentiates and normalizes the raw scores for each class. This normalization maps the class scores to a probability distribution over the range [0, 1], reflecting the likelihood of each class. The computation of these probabilities is typically expressed as follows in Equation (1) [34]. In the equation, x_i represents the raw score for the (i)-th class in the input, while f(x_i) denotes the probability that the sample belongs to the (i)-th class. By employing both traditional machine learning methods and deep learning algorithms, effective classification and identification of various types of paper breakage faults have been achieved. Results indicate that deep learning models outperform traditional machine learning methods across multiple evaluation metrics. However, there is still room for improvement in the performance of deep learning models on certain operating condition datasets. Among the traditional machine learning methods, the random forest classifier performs better than logistic regression and support vector machines. Experimental results show that a fault diagnosis model trained on data from a specific operating condition performs poorly. It struggles to generalize to faults under different operating conditions. This indicates that models trained under single operating conditions have limited generalization capabilities. They cannot be directly applied to other production processes. To address this issue, new methods need to be introduced to enhance the accuracy of fault diagnosis, which is of significant importance for practical applications in paper breakage fault detection. Figure 1 shows the main research framework of the modling section proposed in this study. 2.2.2. Transfer Learning In the production process of household paper, different specific paper products and varying production quantities involve different process parameters. Typically, research on industrial process fault diagnosis is based on having sufficient data from the papermaking process, and fault diagnosis is achieved through a comprehensive analysis of data and empirical mechanisms for different production processes [35]. However, in real industrial processes, faults are low-probability events and process conditions vary under different operating conditions, which can result in situations where sufficient data for fault diagnosis research cannot be collected under certain conditions. In cases with limited data, transfer learning offers a new approach to address diagnosis challenges. In many tasks within machine learning and deep learning, it is generally assumed that data follows the same distribution and comes from the same feature space. Still, in reality, these conditions are often not fully met. Issues such as limited labeled training samples or changes in data distribution may arise. Transfer learning methods are introduced to address these problems, where deep learning can be used for feature extraction, and transfer learning can be used for knowledge transfer. Combining these two approaches, deep transfer learning methods can apply features learned in the source domain to learning tasks in the target domain. This paper combines fault diagnosis models with transfer learning theory to achieve cross-condition fault diagnosis. By extracting features and transferring model parameters trained under data-rich conditions to new conditions, the generalizability and applicability of fault diagnosis models can be enhanced. This study employs two different transfer methods: parameter transfer and feature transfer. For fault diagnosis based on parameter transfer, the model training process includes several parts: source model training, model transfer, model fine-tuning, and model application. The fault diagnosis model based on feature transfer consists of two parts: model training on source domain data and model application testing on target domain data. Different evaluation metrics are introduced to assess the performance of the target domain test samples under various transfer tasks. The main modeling process is shown in Figure 2.

Parameter Transfer Model

To study the fault diagnosis of paper machine paper breakages under different operating conditions, this section establishes a parameter transfer model. The overall framework of the model includes a feature extraction layer and a fault classification layer [36]. The parameters of the feature extraction module are trained using source domain data, and these parameters and weights are frozen. At the same time, the classification layer’s structure parameters are adjusted using samples from the target domain. The fine-tuned model is then used for fault diagnosis under the target conditions. The general process involves training the model on the source domain, applying the model to some labeled data from the target domain for fine-tuning, and obtaining the transfer diagnosis model. In the established parameter transfer learning model, we set several key parameters to construct and train the model. The number of neurons in the model’s hidden layer is set to 16, which helps the model learn complex features. The num_classes is set to 5, as this study addresses a multi-classification problem with five different categories. We set the number of training epochs, num_epochs, to 100 to ensure the model has sufficient learning time. Finally, the batch_size is set to 4, meaning that 4 samples will be used to update the model parameters each time, which helps improve training efficiency and reduce memory consumption. The model structure is shown in Figure 3. Parameter transfer involves applying model parameters trained on a source task to another task to accelerate learning and improve performance. During the transfer, the feature extraction layers or parts of a pre-trained model are used for feature extraction in the target task, with these parameters frozen and not updated during training. Fine-tuning involves further training the model on the target task, usually with a small number of samples, to enhance performance. Despite differences in process parameters under various conditions, data features and parameters from the same machine still share similarities. By extracting common knowledge through the feature extraction network and fine-tuning the model, parameter transfer can be effectively achieved. Incorporate source domain data and small samples from the target domain into the classification network, maintaining the same network structure and parameters as previously constructed. The diagnostic framework is illustrated in Figure 4. In the source model training phase, operational data and paper break fault labels from a specific condition are utilized as source domain data for model training. In contrast, data from the target condition serves as the target domain dataset. A subset of this data is reserved for fine-tuning, with the remainder used to evaluate model performance. During the model transfer phase, parameters from the feature extraction module of the source model are transferred to the target network model to capture target domain features. The classification layer of the initialized target model is then employed for fault classification, thus constructing the parameter transfer fault diagnosis model. In the model fine-tuning phase, the feature extraction layers of the target model are frozen, and the classification layer parameters are refined using the fine-tuning samples. Finally, the fine-tuned model is applied to test samples from the target domain to assess diagnostic efficacy.

Feature Transfer Model

The feature-based transfer method minimizes the distribution difference between the source and target domains while updating network parameters to learn domain-invariant features. To validate the feasibility of domain adaptation methods in cross-condition paper breakage fault diagnosis, the feature transfer method is integrated with the fault diagnosis model mentioned earlier to construct a feature transfer-based diagnostic model. In this study, the feature transfer model updates its parameters using 4 samples at a time during training, with the batch_size set to 4. Next, we use the resampled feature and label data, applying the train_test_split function to divide the dataset into training and testing sets, with test_size = 0.3 indicating that 30% of the data is designated for testing. In comparison, the remaining 70% is used for training. This division helps evaluate the model’s performance and prevents overfitting. During training, num_epochs is set to 100, and in each epoch, the model enters training mode, iterating through the entire target dataset in batches. Each batch's features and labels are extracted from the source and target datasets. The optimizer’s gradients are then cleared to prepare for calculating new gradients. Subsequently, the model propagates forward on the source and target data to generate outputs. The Maximum Mean Discrepancy (MMD) loss is calculated to compare the distributions of the source and target outputs. In contrast, the cross-entropy loss function is used to assess the classification performance of the target output. The final overall loss is the sum of these two loss components. After updating the model parameters through backpropagation, the loss value for each batch is accumulated. Every 50 epochs, the program prints the current training epoch and loss value to monitor the model's training progress. The process is shown in Figure 5. Domain adaptation is one of the primary methods in feature transfer learning. It applies to situations where the sample feature spaces of the source and target domains are the same, but their probability distributions differ. During the evaluation phase, the model uses a context manager to disable gradient calculations, which reduces memory usage and speeds up computation. The test data from the target domain is preprocessed and input into the model as tensors for prediction. The model outputs the predicted probabilities for each category, and by selecting the category corresponding to the maximum probability for each sample, prediction labels are generated. Next, the accuracy between the source and target domains is calculated. Additionally, the macro recall and macro precision are computed, providing a more comprehensive reflection of the model’s performance across different categories. Source domain data is typically used to train the model, while target domain data is used to validate the model’s generalization ability in a new environment. These evaluation metrics help understand the performance differences and effectiveness of the model between the source and target domains. The core idea is to map the data features from different domains to a common feature space, thereby enabling the use of data from other domains to enhance the training of the target domain. A schematic of domain adaptation is shown in Figure 6 [37]. To effectively perform domain adaptation, it is essential to accurately measure the differences between the source and target domains [38]. Adaptive layers in deep learning methods can achieve adaptive data matching from the source and target domains. This adaptation process brings the data distributions of the source and target domains closer together, which can improve the network’s classification performance. The choice of adaptation method has a decisive impact on the model’s generalization ability. Distance functions such as KL divergence, Mahalanobis distance, MMD, and Multiple Kernel Maximum Mean Discrepancy (MK-MMD) can be used for this measurement [39]. As a commonly used metric, MMD maps vectors to a Reproducing Kernel Hilbert Space (RKHS) and calculates the distribution distance between two probability distributions P and Q, in that space. For sample sets X and Y generated from P and Q, respectively, the MMD distance reflects the expected difference between X and Y when mapped to the RKHS H. The computation formula for MMD is given by Equation (2) [40]: In Equation (2), x_i denotes the (i)-th sample in the sample set (X), and y_j denotes the (j)-th sample in the sample set (Y), with (m) and (n) representing the number of samples in sets (X) and (Y), respectively. ϕ(·) denotes the feature mapping function. Based on deep feature transfer theory, this section introduces different distance metric methods to map datasets from different working conditions to a common feature space, achieving similar distributions of source and target domain data within this feature space. The research focuses on cross-condition paper breakage fault diagnosis, with the specific algorithmic process shown in Figure 7, which includes two modules: model training and model application. 2.2.3. Federated Learning Previous studies addressed the issue of having partially labeled data in large sample conditions. In practical papermaking processes, situations with insufficient data samples, make it challenging for data-driven methods to meet fault classification requirements in small sample cases. To address this, centralized processing of datasets from different operating conditions was used to enhance the generalization ability of the network model, aiming to provide a feasible solution for fault diagnosis with small samples. Federated Learning aggregates knowledge from various clients to a central cloud server, where clients jointly train to improve model classification accuracy [41]. Federated Learning features local computation and model transmission. To enhance the efficiency of model training and improve the security of data transmission, this chapter employs model compression on top of Federated Learning. Specifically, only a subset of parameters is uploaded during parameter transmission to increase model transfer efficiency and protect data privacy [42]. Considering the variations in production processes across different locations, the method aims to better extract parameter information from various devices and conditions and to compare the effectiveness of different models. Locally, both fully connected neural networks and convolutional neural networks are used to extract features from each subset of data, and the trained model parameters are uploaded to the cloud [43]. At the cloud server, parameters are aggregated according to various aggregation strategies, resulting in a Federated Learning fault diagnosis model through multiple iterations of parameter uploads and updates [44]. The final Federated global fault diagnosis model is applied to a small sample fault diagnosis. Based on Federated Learning theory, a Federated Learning-based method for diagnosing paper breakage faults is designed, with the detailed process shown in Figure 8. The procedure for constructing a global fault diagnosis model using Federated Learning involves several steps: Initially, datasets from paper machine operations under various conditions are designated as local datasets for each participant, facilitating the development of these local datasets [45]. Subsequently, each local dataset is employed to train local network models, creating multiple local fault diagnosis models. The parameters from these local models are then uploaded to a central server, where the server aggregates the received model updates using various aggregation algorithms to produce a new global model. This updated global model is redistributed to all clients for further local training, continuing until the maximum iteration limit is achieved. To investigate the effectiveness of Federated Learning across different scenarios and assess model performance under various deep learning frameworks, this chapter constructs fault diagnosis models with diverse neural network architectures at each local site [46]. Data under conditions A, B, and C during local training are partitioned into training and testing sets. The training set is utilized for local model training, while the testing set, together with small sample condition data, is employed to evaluate the performance of the Federated global model. For each local training dataset, classification models with fully connected neural networks and convolutional neural network architectures are developed and trained locally [47]. The model parameters are then uploaded to a central server to complete the training of the federated global model. This approach aims to explore the fault diagnosis accuracy of the federated global model under various local model configurations and parameter aggregation strategies. To ensure the validity of comparative experiments, parameters such as activation functions, optimization algorithms, learning rates, local iteration counts, and federated iteration counts are kept consistent [48]. The Federated Averaging (FedAvg) algorithm and Federated Proximal Gradient Descent (FedProx) algorithm are used for aggregation. The FedAvg algorithm aggregates model parameters through weighted averaging. The core idea is to upload local model parameters to the central server, where the server computes the average of all model parameters and returns this average to all local models [49]. The global model can aggregate the parameters of local models, using local data from all devices to train the global model, which can enhance the model’s accuracy and generalization performance. 2.3. Evaluation Metrics In the field of deep learning fault diagnosis [50], for classification problems, commonly used evaluation metrics to assess model performance include Accuracy, Precision, Recall, and the harmonic mean of Precision and Recall (F1-score). The relevant calculations are shown in Equations (3)–(6). In the equations, TP (true positive) is the number of correctly identified positive samples; FP (false positive) is the number of incorrectly identified negative samples; TN (true negative) is the number of correctly identified negative samples; FN (false negative) is the number of missed positive samples. 2.4. Advantages Compared to Other Literature Due to the presence of some small sample conditions in the data, which do not meet the modeling needs, this paper explores the feasibility of federated learning for papermaking industry process fault diagnosis with small sample data. Compared to other single-condition fault diagnosis models, this study has the following advantages: (1) Local models were established using fully connected neural networks and convolutional neural networks to extract relevant features under different conditions. Training and testing were performed using data from various conditions. FedAvg and FedProx aggregation strategies were employed and compared in the parameter aggregation process. Results showed that CNNs performed better in feature learning than fully connected neural networks. FedProx, an improved parameter aggregation strategy based on FedAvg, achieved better diagnostic results. Overall, federated learning methods can learn feature parameters from each model by aggregating different local models. Although the federated global model’s performance on local test samples is slightly inferior to the isolated data model, it still shows comparable performance to the condition data used in modeling with small sample data. This indicates that federated learning can offer a new diagnostic method for small sample conditions. (2) Under the federated learning framework, model compression can improve data protection and transmission efficiency during transfer. Experimental results under different model compression rates indicate that an appropriate compression rate can effectively reduce communication overhead and enhance data privacy protection while maintaining a certain level of diagnostic accuracy.