Machine Learning Methods

Deep Neural Networks, Support Vector Machines,

ARTMAP and Bayesian Classifiers

Comparative Study of Machine Learning Methods on Circle-In-Square and MNIST Tasks
March 9, 2025
Lloyd Watts

In 2025, Artificial Intelligence (AI) research and product development is dominated by Large Language Models (LLMs), which in turn are based on Auto-Regressive Decoder-Only Transformers, which are based on Deep Neural Networks (DNNs). DNNs are the dominant core component because they are powerful, universal function approximators with a well-defined learning rule (Back-Propagation), and their high performance and scalability to large and deep networks has led to a decade of industrial infrastructure development, in software (Pytorch, TensorFlow), and in hardware (GPUs with CUDA software development platform, multi-core CPUs).

But DNNs have a number of serious problems, including lack of interpretability/explainability, training requires multiple passes through the entire training dataset, no few-shot learning, and no online continuous learning without catastrophic forgetting. And there are other Machine Learning methods which have different/better properties: Bayesian Classifiers, Adaptive Resonance Theory (ARTMAP), Nearest Neighbor Classifier, Support Vector Machines, 2-Layer DNNs, Convolutional DNNs, DNNs with Neocortix Deep Attribution Network.

The purpose of this investigation was to compare all of the above Machine Learning methods on two familiar tasks: the Circle-In-Square Task and the MNIST Handwritten Digit Recognition Task. We measured their training time and test time on a 64-core CPU machine with 128 GB of RAM, and we created plots and visualizations of the data sets and the internal data representations and decision boundaries of each model, to illustrate why some models are explainable and some are not.

Training Datasets for the Circle-In-Square and MNIST Tasks

Figure 1: Training Datasets for the Circle-In-Square and MNIST Tasks.
The Circle-In-Square dataset has 1000 data points, in two classes, in two dimensions. The MNIST dataset has 60,000 samples, in 10 classes (digits 0-9), in 28x28=784 dimensions. We're showing 8 representative samples for each class, above.

We are using the Circle-In-Square dataset because the whole dataset and any model's decision boundaries can be visualized easily. We are using the MNIST dataset because it is a well-known dataset which established Deep Convolutional Neural Networks as the best-performing model for image classification tasks, and thus it is a de facto standard reference for evaluating other models. But with 60,000 samples in 10 classes in 784 dimensions, it is a challenge to visualize the MNIST data and the model decision boundaries.

Summary of Results for the Circle-In-Square Task

Figure 2: Model Results for the Circle-In-Square Task. In Figure 2, we can see how the different models work, and what accuracy they can get.

Fuzzy ARTMAP with Vigilance rho=0.95 gets accuracy=97.40% with 250 Learned Categories. This method has many desirable properties: It is inherently explainable, it can do single-pass learning, few-shot learning, online and continuous learning, with no catastrophic forgetting.
Fuzzy ARTMAP with Vigilance rho=1.0 gets accuracy=97.33% with 1000 Learned Categories.
Euclidean Nearest Neighbor Classifier gets accuracy=97.40%.
Bayesian Classifier gets accuracy=94.94%.
Support Vector Machine with Radial Basis Function Kernel gets accuracy=99.00%. This is the best-performing model on the Circle-In-Square task.
Deep Neural Network gets accuracy=98.63%. This is the second-best-performing model on the Circle-In-Square task.

But what is the relationship between the neurons and nonlinear activation functions and the two decision surfaces for the Deep Neural Network? Figure 3 shows the Pytorch code for this two-layer fully-connected neural network with 2 inputs (2D input vectors), 1700 hidden neurons, and 2 outputs (2 classes). It uses a Rectified Linear or ReLU() activation function, also shown in Figure 3.

Figure 3: DNN Pytorch code and Rectified Linear ReLU() Activation Function.

In Figure 4, we show that the sum of 4 bent planes can make a piecewise planar approximation to a smooth curved surface.

Figure 4: A tiny Neural Network with 4 hidden neurons can create 4 Bent Planes.

In the Circle-In-Square example, each of the 1700 hidden neurons creates a bent-plane function.

Figure 5: DNN individual neuron bent-plane crease lines, and the composite decision surfaces created from weighted sums of 1700 rotated bent planes.
Now we can see why Explainability/Interpretability of Deep Neural Networks is considered a difficult problem.

Finally, we note that Deep Neural Networks are the dominant Machine Learning method, and yet they are not inherently explainable, so significant effort is going into finding ways to make them explainable. But we should remember that Fuzzy ARTMAP is also a high-performance Machine Learning method, and it is inherently explainable, because it carries representative training set data with the Learned Categories in its model representation.

Now that we have examined the different Machine Learning models on the simple 2D 2-class Circle-In-Square problem, let's look at the more complex 784-D 10-class MNIST problem.

Summary of Results for the MNIST Task

The MNIST Handwritten Digit task consists of 28x28-pixel images of 10 handwritten digits (0-9). There are 60,000 samples in the training set, and 10,000 samples in the test set. 8 representative training set samples in each of the 10 classes are shown in Figure 1.

We used high-performance implementations of all of the above Machine Learning models, and evaluated them on their recognition accuracy, number of stored parameters in the trained model, training duration, test duration, number of training epochs (training passes through the training set), and binary judgment of online continuous learning, and inherent explainability. All of the Machine Learning models were trained and run on a 64-core CPU machine with 128 GB of RAM,

Figure 6: Model Results for the MNIST Task.

We can make some broad statements about the different models and their performance on the MNIST task:

Accuracy: Deep Convolutional Neural Networks get the highest accuracy=99.2%. Of the models that are inherently explainable, Euclidean Nearest Neighbor gets the highest accuracy=96.9%. Bayes gets the lowest accuracy=94.7%.
Number of Parameters: Bayes has the lowest number of stored parameters=615,400. The two high-accuracy explainable methods are Euclidean Fuzzy ARTMAP and Deep Convolutional Network with 15.4 Million and 20.4 Million parameters, respectively.
Training Duration and Training Epochs: Manhattan and Euclidean Nearest Neighbors have zero training duration, Bayes has extremely fast training duration (0.5s), Fuzzy ARTMAP has very fast training duration (19s), and Deep Convolutional Neural Networks take 4.4 minutes over 14 training epochs.
Online Training: The models capable of Online Training are Bayes, Manhattan and Euclidean Nearest Neighbor, and Fuzzy ARTMAP.
Explainability: The Explainable models are Bayes, Manhattan and Euclidean Nearest Neighbor, Fuzzy ARTMAP, and Deep Neural Network with Neocortix Deep Attribution Network.

Now, let us look closer at the individual models.

Fuzzy ARTMAP with Vigilance rho=0.905 gets accuracy=96.6% with 19,683 Learned Categories. This method has many desirable properties: It is inherently explainable, since the Learned Categories are directly related to the training samples.
Manhattan and Euclidean Nearest Neighbor Classifiers get accuracy=96.3% and 96.9%, respectively.
Bayesian Classifier gets accuracy=94.7%.
Support Vector Machine with Radial Basis Function Kernel gets accuracy=98.36%.
Deep Convolutional Neural Network gets accuracy=99.2%.
Deep Convolutional Neural Network with Neocortix Deep Attribution Network gets accuracy=99.2%, and is Explainable.

Conclusions

We have examined a number of popular Machine Learning methods, including Deep Neural Networks, Support Vector Machines, Adaptive Resonance Theory (ARTMAP), Bayesian Classifier, Manhattan and Euclidean Nearest Neighbor Classifiers, and Neocortix Deep Attribution Networks. We have applied those models to the simplest 2-dimensional 2-class Circle-In-Square problem, and the more complex 784-dimensional 10-class MNIST Handwritten Digits problem. We have shown how to visualize the training data, decision surfaces and decision boundaries of each method in the Circle-In-Square problem. We have measured the performance of each method on the MNIST problem, including accuracy, training and test duration, explainability, and capacity for online continuous learning. We have shown that when Explainability is achieved, it comes at a cost of additional parameter storage and training time.