Authors: Antoni Olbrysz, Karol Struniawski, Tomasz Wierzbicki
of Contents
- Introduction
- New Dataset of Pollen Images
- Extraction of Individual Pollen Images
- Classification of Individual Pollen Images
- Conclusions
- Acknowledgement
1. Introduction
Pollen classification is an interesting domain in visual image recognition, with a broad range of use cases across ecology and biotechnology, such as studies of plant populations, climate change, and pollen structure. Despite this, the subject is relatively unexplored, as few datasets have been composed of images of such pollens, and those that exist are often lackluster or otherwise insufficient for the training of a proper visual classifier or object detector, especially for images containing mixtures of various pollens. Besides providing a sophisticated visual identification model, our project aims to fill this gap with a custom-made dataset. Visual pollen classification is often complicated to solve without machine vision, as modern biologists are often incapable of differentiating between pollen of different plant species based on images alone. This makes the task of quickly and efficiently recognising harvested pollens extremely challenging, provided the pollen particles’ source is unknown beforehand.
1.1 Available Datasets of Pollen Images
This section highlights the parameters of several freely available datasets and compares them to the properties of our custom set.
Dataset 1
Link: https://www.kaggle.com/datasets/emresebatiyolal/pollen-image-dataset
Number of classes: 193
Images per class: 1-16
Image quality: Separated, clear images, sometimes with labelings
Image colour: Various
Notes: The dataset seems composed of incongruent images taken from multiple sources. While broad in classes, each contains only several photos, insufficient for training any image detection model.
Dataset 2
Link: https://www.kaggle.com/datasets/andrewmvd/pollen-grain-image-classification
Number of classes: 23
Images per class: 35, 1 class 20
Image quality: Well separated, slightly blurry, no text on images
Image colour: Uncoloured, consistent
Notes: Localised, well-prepared dataset for the classification of Brazilian Savannah pollen. Consistent image source, yet the number of images per class may pose issues when aiming for high accuracy.
Dataset 3
Link: https://www.kaggle.com/datasets/nataliakhanzhina/pollen20ldet
Number of classes: 20
Images per class: Exorbitant
Image quality: Self-explanatory images, with separated and joined pollen images.
Image colour: Dyed, consistent
Notes: An insurmountable amount of well-labeled, consistent, and high-quality images make this dataset of the highest quality. However, the colouring present may be an issue in specific applications. Additionally, the magnification and ability of the pollens to intersect may pose problems in mixed-pollen scenarios.
2. New Dataset of Pollen Images
Our dataset is a collection of high-quality microscope images of 4 different classes of pollens belonging to common fruit plants: the European gooseberry, the haskap berry, the blackcurrant, and the shadbush. These plant species have not been part of any previous dataset, so our dataset contributes new data towards visual pollen classification.
Each class contains 200 images of multiple grains of pollen, each image without dye. It was obtained in collaboration with the National Institute of Horticultural Research in Skierniewice, Poland.
Number of classes: 5 (4 pollens + mixed)
Images per class: ~200
Image quality: Clear images, images contain multiple pollen fragments, mixed images present
Image colour: Undyed, consistent
Our dataset focuses on locally available pollens, class balance, and an abundance of images to train on without added dye, which may make the classifier unsuitable for some tasks. Additionally, our proposed solution contains images with mixtures of different pollen types, aiding in training detection models for field-collection applications. Exemplary images from the dataset are represented as Figures 1-4.
The full dataset is available from the corresponding author on reasonable request. The data acquisition steps are composed of the sample preparation and taking microscopic images, which were prepared by Professor Agnieszka Marasek-Ciołakowska and Ms. Aleksandra Machlańska from the Department of Applied Biology at The National Institute of Horticultural Research, for which we are very thankful. Their efforts have proven invaluable for the success of our project.
We first extracted images of individual pollens from the photos in our datasets to train various models to recognize pollens. Each of those photos contained multiple pollens and other lifeforms and pollution, making identifying the pollen species far more difficult. We used the YOLOv12 model, a cutting-edge attention-centric real-time object detection model developed by Ultralytics.
3.1 Fine-Tuning YOLOv12
Thanks to YOLOv12’s innovative nature, it can be trained even on tiny datasets. We experienced this phenomenon firsthand. To prepare our own dataset, we manually labeled the pollens’ locations on ten images in each of the four classes of our dataset using CVAT, later exporting the labels into .txt files corresponding to individual images. Then, we organized our data into a YOLOv12-appropriate format: we divided our data into a training set (7 image-label pairs per class, in total 28) and a validation set (3 image-label pairs per class, in total 12). We added a .yaml file pointing towards our dataset. It can be noticed that the dataset was truly very small. The resulting image in prediction mode with detected individual pollens with the confidence overlay is represented as Fig. 5. We also downloaded the model (YOLO12s) from the YOLOv12 website. Then we started the training.
The model proved to detect pollens with very high accuracy, but there was one more thing to consider: the model’s confidence. For every detected pollen, the model also outputted a value of how specific its prediction is. We had to decide whether to use a lower threshold for confidence (more images, but higher risk of malformed or non-pollen photos) or a higher one (fewer images, but lower chance of non-pollen). We eventually settled on trying out two thresholds, 0.8 and 0.9 to evaluate which one would work better when training classification models.
3.2 Exporting the individual-pollen datasets
To do this, we launched the model’s prediction on all of the class-specific images in our dataset. This worked very well, but after exporting, we encountered another issue — some photos were cropped, even on higher thresholds. For this reason, we added another step before exporting our individual pollens: we eliminated images with a disproportionate aspect ratio (see example as Fig. 6). Specifically, 0.8, dividing the smaller dimension by the larger one.
Then, we resized all the images into 224×224, the standard dimension for input images for deep learning models.
3.3 Individual-pollen datasets — a short analysis
We ended up with two datasets, one made with a confidence threshold of 0.8 and the other with 0.9:
- 0.8 Threshold:
- gooseberry — 7788 images
- haskap berry — 3582 images
- blackcurrant — 4637 images
- shadbush — 4140 images
Total — 20147 images
- 0.9 Threshold:
- gooseberry — 2301 images
- haskap berry — 2912 images
- blackcurrant — 2438 images
- shadbush —1432 images
Total — 9083 images
A quick look at the numbers shows that the 0.9 threshold dataset is over twice as small as the 0.8 threshold one. Both datasets are not balanced — the 0.8 one due to gooseberry and the 0.9 one due to shadbush.
YOLOv12 was an effective tool for segmenting our images into two single-pollen image datasets, even though we encountered some difficulties. The newly created datasets may be unbalanced, yet their size should compensate for this drawback, mainly since every class is widely represented. There is a lot of potential in them for future training of classification models, but we will have to see for ourselves.
4. Classification of Individual Pollen Images
4.1 An Overview of Model Rating Metrics
One must devise metrics to measure performance to properly approach training models, whether classical ones working on statistical features, or more complex approaches, such as convolutional neural networks or vision transformers. Through the years, many methods have been devised for accomplishing these tasks — from statistical measures such as F1, precision, or recall, to more visual metrics such as GradCAM, that allow a deeper insight into the model’s inner workings. This article explores the grading methods used by our models, without going into unnecessary detail.
Recall
Recall is described as the ratio of accurate guesses of one class to the total guesses of that class (see Eq. 1). It measures what percentage of images marked as a class belong to it. Working on separate classes makes it functional in both balanced and imbalanced datasets.
Eq. 1— Formula for recall.
Precision
As opposed to recall, precision is the percentage of correctly marked items among all items belonging to the class (see Eq. 2). It measures the percentage of items in a class that were guessed correctly. This metric performs similarly to recall.
Eq. 2 — Formula for precision.
F1 Score
The F1 Score is simply the harmonic mean of precision and recall (see Eq. 3). It helps combine precision and recall into a concise measurement. Hence, it still performs excellently even on unbalanced datasets.
Eq. 3 — Formula for F1.
Confusion Matrix
The confusion matrix is a visual measure comparing the number of guesses made for one class to the actual number of images in this class. It helps to illustrate mistakes made by the model, which may have trouble with only specific courses (see Fig. 7).
GradCAM
GradCAM is a measure of CNN performance that visualises which areas of the image influence the prediction. To do this, the method computes the gradients from 1 convolutional layer and determines an activation map that visually layers on top of the image. It greatly aids in understanding and explaining the model’s “reasons” for labelling a particular image as a specific class (see example in Fig. 8).
These metrics are only a few of a vast sea of measurements and visualisation methods used in machine learning. Yet, they have proven sufficient for measuring the performance of the model. In further articles, metrics will be brought up accordingly as new classifiers are used and introduced in the project.
4.2 Individual Pollen Classification with Standard Models
With our images preprocessed, we could move on to the next stage: classifying individual pollen into species. We tried three approaches - standard, simple classifiers based on features extracted from images, Convolutional Neural Networks, and Vision Transformers. This article outlines our work on standard models, including the kNN classifier, SVMs, MLPs, and Random Forests.
Feature extraction
To make our classifiers work, we first had to obtain features on which they could base their predictions. We settled for two main types of features. One was statistical measures based on the presence of pixels with a particular colour (one from the RGB model) for a specific image, such as the mean, standard deviation, median, quantiles, skew, and kurtosis — we extracted them for every color layer. The other was GLCM (Gray Level Co-Occurrence Matrix) features: contrast, dissimilarity, homogeneity, energy, and correlation. Those were obtained from grayscale-converted images, and we extracted each at different angles. Every single image had 21 statistical features and 20 GLCM-based features, which amounts to 41 features per image.
k-Nearest-Neighbors (kNN)
The kNN is a classifier that uses a spatial representation of data to classify data by detecting the k nearest neighbours of a feature to predict its label. The said classifier is fast, yet other methods outperform it.
kNN Metrics:
0.8 Dataset:
F1: 0.6454
Precision: 0.6734
Recall: 0.6441
0.9 Dataset:
F1: 0.6961
Precision: 0.7197
Recall: 0.7151
Support Vector Machine (SVM)
Like the kNN, the SVM represents data as points in a multi-dimensional space. Still, instead of finding nearest neighbours, it tries algorithmically separating the data with a hyperplane. This yields better results than the kNN, but introduces randomness and is still outclassed by other solutions.
SVM Metrics:
0.8 Dataset:
F1: 0.6952
Precision: 0.7601
Recall: 0.7025
0.9 Dataset:
F1: 0.8556
Precision: 0.8687
Recall: 0.8597
Multi-Layered Perceptron (MLP)
The Multi-Layered Perceptron is a model inspired by the human brain and its neurons. It passes inputs through a network of layers of neurons with their own individual weights, which are altered during training. When well-optimized, this model can sometimes achieve great results for a standard classifier. Still, pollen recognition was not one of them - it performed poorly compared to other solutions and was not consistent.
MLP Metrics:
0.8 Dataset:
F1: 0.8131
Precision: 0.8171
Recall: 0.8173
0.9 Dataset:
F1: 0.7841
Precision: 0.8095
Recall: 0.7940
Random Forest
The random forest is a model famous for its explainability - it is based on decision trees, which classify data based on thresholds, which humans can analyze far more easily than, for instance, weights in neural networks. The Random Forest performed fairly well and consistently - we found that 200 trees was optimal. Still, it was outclassed by more complex classifiers.
RF Metrics:
0.8 Dataset:
F1: 0.8211
Precision: 0.8210
Recall: 0.8233
0.9 Dataset:
F1: 0.8150
Precision: 0.8202
Recall: 0.8216
The classical models exhibited varied performance levels- some performed worse than expected, whereas others delivered fairly good metrics. However, this is not yet the end. We still have advanced deep learning models to try out, such as Convolutional Neural Networks and Vision Transformers. We expect that they will perform significantly better.
4.3 Individual Pollen Classification with Convolutional Neural Networks
Classical models such as MLPs, Random Forests, or SVMs in individual pollen classification yielded mediocre to pretty good results. However, the next approach we decided to try was Convolutional Neural Networks (CNNs). They are models that generate features by processing images and are known for their effectiveness.
Instead of training CNNs from scratch, we used a transfer learning technique — we took pre-trained models, specifically ResNet50 and ResNet152, and fine-tuned them to our dataset. This approach makes training significantly faster and less resource-demanding. It also allows for far more effective classification due to the models’ already being professionally trained on large datasets. Before training, we also had to normalize the images.
In terms of metrics, we used Grad-CAM, a method that attempts to highlight the areas of an image that influenced a model’s prediction the most, in addition to standard metrics such as F1 score, precision, and recall. We also included confusion matrices to see if our CNNs struggle with any particular class.
ResNet50
ResNet50 is a CNN architecture developed by Microsoft Research Asia in 2015, which was a significant step towards creating far deeper and more efficient neural networks. It is a residual network (hence the name ResNet) that uses skip connections to allow direct data flow. This, in turn, mitigates the vanishing gradient problem.
We expected this model to perform worse than ResNet152. Our expectations were quickly subverted as the model delivered predictions on the same level as ResNet152 on both datasets, as represented in the below-listed metrics and confusion metrics (see Fig. 9 and Fig. 10), as well as Grad-Cam visualization (see Fig. 11).
ResNet50 Metrics:
0.8 Dataset:
F1: 0.98
Precision: 0.98
Recall: 0.98
0.9 Dataset:
F1: 0.99
Precision: 0.99
Recall: 0.99
Regarding Grad-CAM, it did not provide any valuable insights about the model’s inner workings - the highlighted zones included the background and seemingly random places. Because it achieves very high accuracy, the network appears to notice patterns undetectable by the human eye.
ResNet152
Also a development of Microsoft’s researchers, the ResNet152 is a residual network and a CNN architecture with significant depth and deep learning capabilities far exceeding those of the ResNet50.
Therefore, our expectations for this model were higher than for ResNet50. We were disappointed to see that it performed on par with it. It performed excellently (see Fig. 12 and Fig. 13 with confusion matrices and Fig. 14 with Grad-Cam visualizations).
ResNet152 Metrics:
0.8 Dataset:
F1: 0.98
Precision: 0.98
Recall: 0.98
0.9 Dataset:
F1: 0.99
Precision: 0.99
Recall: 0.99
Grad-CAM was not helpful for ResNet152 either - we experienced the enigmatic nature of deep learning models, which achieve high accuracy but cannot be explained easily.
We were surprised that the more complex ResNet152 did not outperform the ResNet50 on the 0.9 dataset. Both achieved the highest metrics out of any models we have tried so far - they trumped the classical models, with the difference between the best classical model and the CNNs exceeding 10 percentage points. It is time to test the most innovative model - the Vision Transformer.
4.4 Individual Pollen Classification with Vision Transformers
For individual pollen classification, we tried out simple models, which provided a varied level of performance, from insufficient to satisfactory. Then, we implemented convolutional neural networks, which completely trumped their performance. Now it is time for us to try out the innovative model known as the Vision Transformer.
Transformers, in general, originate from the famous 2017 paper “Attention Is All You Need” by researchers at Google, but they were initially used mainly for natural language processing. In 2020, the transformer architecture was applied in computer vision, yielding the ViT — Vision Transformer. Its excellent performance marked the beginning of the end for Convolutional Neural Networks’ reign in the area.
Our approach here was similar to what we used when training CNNs. We imported a pre-trained model: vit-base-patch16–224-in21k, a model trained on ImageNet-21k. Then, we normalized our dataset images, fine-tuned them, and noted down the results of metrics and confusion matrices (see Fig. 15 and Fig. 16).
vit-base-patch16–224-in21k results:
0.8 Dataset:
F1: 0.98
Precision: 0.98
Recall: 0.98
0.9 Dataset:
F1: 1.00
Precision: 1.00
Recall: 1.00
In the 0.8 dataset, the Vision Transformer presented a level of performance that did not exceed that of the Residual Networks, and it struggled with similar problems — it misclassified Gooseberry as Blackcurrant. However, on the 0.9 dataset, it achieved a nearly perfect score. We witnessed innovation triumph over more dated solutions, which urged us to save the model and designate it as our model of choice for more demanding tasks.
4.5 Comparison of Metrics for Various Models
For our pollen classification tasks, we have used many models: traditional models, including the kNN, SVM, MLP, and Random Forest; Convolutional Neural Networks (ResNet50 and ResNet152), and a Vision Transformer (vit-base-patch16–224-in21k). This article serves as an overview and a performance ranking (see Tab. 1).
Ranking
6. kNN (k-Nearest-Neighbors)
The simplest classifier. As expected, it trained quickly, but performed the worst.
5. MLP (Multi-Layered Perceptron)
The model’s architecture is based on the human nervous system. The MLP was outperformed by other standard models, which we did not expect.
4. RF (Random Forest)
The Random Forest classifier performed with the highest consistency of all models, but its metrics were far from ideal.
3. SVM (Support Vector Machine)
The unexpected winner among the typical classifiers. Its performance was random but yielded good results for a standard classifier on the 0.9 dataset.
2. ResNet50 and ResNet152 (Residual Networks)
Both architectures achieved the same high results thanks to their complexity, far exceeding the capabilities of any standard classifier on both datasets.
1. ViT (Vision Transformer)
The most innovative solution we have tried trumped the classical models and caught up to the Residual Networks on the 0.8 dataset. Yet the real challenge was the 0.9 dataset, where the CNNs reached an insurmountable accuracy of 0.99. To our surprise, the Vision Transformer’s results were so high that they were rounded to 1.00 - a perfect score. Its results are a true testament to the power of innovation.
Note: the classification report rounded up the model’s metrics - they are not exactly equal to 1, as that would mean that all images without exception were classified correctly. We settled for this value because only a marginal five images (0.27%) were misclassified.
Through comparing different classifiers in the domain of visual pollen recognition, we were able to experience the history and evolution of machine learning personally. We tested models with varying degrees of innovation, starting with the simplest classifiers and the attention-based Vision Transformer, and noticed how their results increased along with their novelty. Based on this comparison, we unanimously elected the ViT as our model of choice for working with pollen.
5. Conclusions
The task of visually classifying pollen, which has eluded biologists around the world and lay outside the grasp of human ability, has finally been proven possible thanks to the power of machine learning. The models presented in our publication have all shown potential to classify the pollens, with varying degrees of accuracy. Some models, such as CNNs or the Vision Transformer, have reached near perfection, recognising pollen with precision unseen in humans.
To better understand why this accomplishment is so impressive, we illustrate it in Fig. 17.
It is highly likely that most readers cannot correctly classify these images into the four classes mentioned previously. On the other hand, our models have proven to recognise them with almost perfect accuracy, reaching a top F1 score of over 99%.
One may wonder what such a classifier could be used for, or why it was trained in the first place. The applications of this approach are numerous, from tracking plant populations to measuring airborne allergen levels on a local scale. We constructed the models to not only provide a tool for palynologists to use to classify pollens they might collect, but also to provide a research platform for other machine learning enthusiasts to build off of, and to demonstrate the ever-expanding applications of this field.
On that note, this is the end of this publication. We sincerely hope the reader finds this helpful information in their research endeavours and that our articles have sparked ideas for projects using this technology.
6. Acknowledgments
We are very thankful to Professor Agnieszka Marasek-Ciołakowska from the National Institute of Horticultural Research, Skierniewice, Poland, for preparing samples and taking microscopic images of them using the Keyence VHX-5000 microscope. The authors possess the complete, non-restricted copyrights to the dataset used in this research and all images used within this article.