Weather Image Classification with Keras
Weather phenomenon recognition notably affects many aspects of our daily lives, for example, weather forecast, road condition monitoring, transportation, agriculture, forestry management, and the detection of the natural environment. In contrast, few studies aim to classify actual weather phenomenon images, usually relying on visual observations from humans.
We are in the process of “democratizing AI”, where tools are being developed and popularized that are aimed to be used by everyone, not just the small academic community. This is where Keras comes in, a higher level (= user friendlier) layer on top of TensorFlow (and other ML frameworks) that allows you to make ML and more specifically Neural Networks (NN) faster and easier.
In this article we study weather classification from images using Convolutional Neural Networks (CNNs) with Keras+Tensorflow. We present a dataset with 3348 images with six weather conditions (rain, shine, sunrise, fogsmog, snow, cloudy) obtaining a precision of 90% in the validation data and 87% in test data. Realizing the automatic and high-quality classification of weather phenomena images can provide a reference for future research on weather image classification and weather forecasting.
Introduction
Nowadays, machine learning is developing rapidly, enabling researchers to apply machine learning to various academic fields. More specifically, if an image recognition ML model could be built to identify conditions by simply looking at images of the weather.
The analysis of weather phenomenon plays a crucial role in various applications, for example, environmental monitoring, weather forecasting, and the assessment of environmental quality
Besides, different weather phenomena have diverse effects on agriculture. Therefore, accurately distinguishing weather phenomena can improve agricultural planning. Furthermore, weather phenomena not only strongly influences vehicle assistant driving systems (by snow, sandstorm, rain, etc.) but also affects us in our daily lives, such as the wearing, traveling, and solar technologies.
What is Image Classification?
Image classification is the process of segmenting images into different categories based on their features. A feature could be the edges in an image, the pixel intensity, the change in pixel values, and many more. We will try and understand these components later on. For the time being let’s look into the images below (refer to Figure 1). The three images belong to the same individual however varies when compared across features like the color of the image, position of the face, the background color, color of the shirt, and many more. The biggest challenge when working with images is the uncertainty of these features. To the human eye, it looks all the same, however, when converted to data you may not find a specific pattern across these images easily.
What is CNN?
CNN or the convolutional neural network (CNN) is a class of deep learning neural networks. In short think of CNN as a machine learning algorithm that can take in an input image, assign importance (learnable weights and biases) to various aspects/objects in the image, and be able to differentiate one from the other.
CNN works by extracting features from the images. Any CNN consists of the following:
- The input layer which is a grayscale or color image
- The Output layer which is a binary or multi-class labels
- Hidden layers consisting of convolution layers, ReLU (rectified linear unit) layers, the pooling layers, and a fully connected Neural Network
In this article we designed a weather phenomenon classification network, which is a deep learning algorithm using a data set containing 3348 images with 6 categories of weather phenomena (rain, shine, cloudy, fogsmog, sunrise, snow).
The remainder of this paper is organized as follows. Section2 and 3 introduces the data we used. Section4 we set parameters to perform image preprocessing and data augmentation. Section5 we define the architecture of our two models with one and two CNN blocks. Section 6 and 7 is the training process and the preliminary results. Section 8 we present some predictions with our two trained models. Section 9 we propose CNN using transfer learning. Section 10 to show final resultas on our test dataset. Finally, Section 11 concludes this article and discusses the outlook of future work.
2. Importing Weather Dataset
Multi-class weather dataset(MWD) for image classification is a valuable dataset used in the research paper entitled “Multi-class weather recognition from still image using heterogeneous ensemble method”. This dataset provides a platform for outdoor weather analysis by extracting various features for recognizing different weather conditions.
URL: https://data.mendeley.com/datasets/4drtyfjtfy/1
To the best of our knowledge, a large number of the labeled data set is necessary when proposing any of the classification models in supervised learning. The quantity and quality of the data set directly affect the classification performance of the model, and the classification accuracy can be significantly improved by creating an extensive database for training, validation and testing.
In addition to the presented dataset, images with similar weather conditions have also been downloaded manually, in this way we are providing to the model with a greater number of samples to improve accuracy.
Complete dataset can be download here:
https://www.kaggle.com/mauricioarancibia/weatherimgclass/download
3. Exploratory Data Analysis (EDA)
As a good practice EDA is important to understand the data first and try to gather as many insights from it. EDA is all about making sense of data in hand,before getting them dirty with it. So, let’s start an exploration of the available data. The dataset structure is composed as follows:
Now it is important to take a look at distribution of samples of the images by each category. In the following figure we can see that for training dataset the category of rain and fogsmog are the ones with the largest number of samples.
Similar distribution is shown in the training dataset (See next figure)
For test dataset, we have few samples for cloudy and shine, we will see later how the trained model peforms with these samples.
Now let’s take a look at some sample images, from top, from left to right: cloudy, fogsmog, rain, shine, snow, sunrise. The images shown have different sizes so they need to be corrected and resized in the next processing image stage.
4. Image Processing and Data Augmentation
In order to make the most of our few training examples, we will “augment” them via a number of random transformations, so that our model would never see twice the exact same picture. This helps prevent overfitting and helps the model generalize better.
In Keras this can be done via the keras.preprocessing.image.ImageDataGenerator class. This class allows you to:
- Configure random transformations and normalization operations to be done on your image data during training
- instantiate generators of augmented image batches (and their labels) via
.flow(data, labels)or.flow_from_directory(directory). These generators can then be used with the Keras model methods that accept data generators as inputs,fit_generator,evaluate_generatorandpredict_generator.
At this stage we are going to generate images using Keras Generator and then save it in a temporary directory “preview/”, in this way we can see how the Data Augmentation strategy is working. We disable rescaling so that images can be easily viewed.
Now it’s time to load the dataset with the image generator from Keras. We will use the following configurations: rescaling, rotation, shear range, zoom and horizontal flip.
5. Model Implementation
The CNN model works in two steps: feature extraction and Classification
Feature Extraction is a phase where various filters and layers are applied to the images to extract the information and features out of it and once it’s done it is passed on to the next phase i.e Classification where they are classified based on the target variable of the problem.
A typical CNN model looks like this:
- Input layer
- Convolution layer + Activation function
- Pooling layer
- Fully Connected Layer
For this work initially and from scratch we will implement two simple CNN models, the first with a one CNN block and the second with two CNN blocks, the reason, we would liek see differences and performance improvements between those. Finally in Section 8 we will implement a model using transfer learning and starting from a pre-trained CNN network. To do this we will be using the VGG16 (Simonyan & Zisserman,2014).
Model 1 Architecture
This One-block model has two single convolutional layer with 32, 16 filters followed by a max pooling layer. Then we define our fully connected layers for classification with one flatten, one dense and one dropout layer, and at last the ouput layer with “softmax” activation function.
Next, we need to compile our model. Compiling the model takes three parameters: optimizer, loss and metrics.
The optimizer controls the learning rate. We will be using ‘adam’ as our optmizer. Adam is generally a good optimizer to use for many cases. The adam optimizer adjusts the learning rate throughout training.
The learning rate determines how fast the optimal weights for the model are calculated. A smaller learning rate may lead to more accurate weights (up to a certain point), but the time it takes to compute the weights will be longer.
We will use ‘categorical_crossentropy’ for our loss function. This is the most common choice for muticlass classification. A lower score indicates that the model is performing better.
To make things even easier to interpret, we will use the ‘accuracy’ metric to see the accuracy score on the validation set when we train the model.
Let’s display our model 1 architecture
Model 2 Archicture
For this second model we propose two block convolutional as follows:
- Block 1: Two 32 filter layers, followed by a max pooling layer.
- Block 2: Two 64 filter layers, followed by a max pooling layer.
- Fully connected: two dense and two dropout layers, and finally the classifier layer with “softmax” activation function.
- For compile parameters we will use the same ones from model1.
Let’s display our model 2 architecture
6. Training Model
Now we will train our model. To train, we will use the ‘fit()’ function on our model with the following parameters: training data , validation data, and the number of epochs.
The number of epochs is the number of times the model will cycle through the data. The more epochs we run, the more the model will improve, up to a certain point. After that point, the model will stop improving during each epoch. For our models, we will set the number of epochs to 50.
Model 1
After finishing the training our first model returns an accuracy of 84.54% for training and 80.30% for validation, not bad for a fairly simple model. And if we observe the last 5 epochs in validation we get some results above 83%. In the next training model we will see how this two CNN block model performs.
Now if we observe the evolution of the training on each epoch both in training and in validation, the behavior of the learning curves for this first model give us good results since there is no over fitting.
Model 2
After finishing the training process, our second model returns an accuracy of 89.43% for training and 81.78% for validation dataset. And if we observe last five epochs we can even get values above 87% for validation.
Now if we look at the learning curve, it shows the same behavior as our first model (One-block). We get better performance values, which means that by adding a second CNN block we can improve our model. Maybe it would be interesting to try more epochs to get a better performance but taking care do not overfit the model.
7. Preliminary Scores
Now it’s time to show our final results from both models and how they perform with test dataset.
Model 1 results
For model 1 we get 82,26% accuracy on test dataset, which is not a bad thing, considering that we use a relatively small dataset.
Model 2 results
For model 2 we get a precision value of 84.82% in the test dataset. This two CNN blocks confirms an improvement in model performance. As mentioned above, it is possible to keep improving this model by increasing epochs as well as including more CNN layers, but having control to avoid overfitting.
Finally, it is interesting to mention the capacity and learning power of the CNNs even with so few samples. Later we will carry out a test using a CNN already trained (Transfer Learning), which will allow us to get even better results than those presented above.
8. Transfer Learning
It can take weeks to train a neural network on large datasets. Luckily, this time can be shortened thanks to model weights from pre-trained models — in other words, applying transfer learning.
Transfer learning is a technique that works in image classification tasks and natural language processing tasks. I
Convolutional Networks are great for image problems however, they are computationally expensive if you use a big architecture and don’t have a GPU. For that, we have two solutions:
The most common incarnation of transfer learning in the context of deep learning is the following workflow:
- Take layers from a previously trained model.
- Freeze them, so as to avoid destroying any of the information they contain during future training rounds.
- Add some new, trainable layers on top of the frozen layers. They will learn to turn the old features into predictions on a new dataset.
- Train the new layers on your dataset.
VGG16
For this work we will use the models based on VGG compared with commonly used transfer models, VGG16 has a simple structure, is easy to train, occupies a small memory, and can prevent over-fitting for small data sets.
- VGG models are a type of CNN Architecture proposed by Karen Simonyan & Andrew Zisserman of Visual Geometry Group (VGG), Oxford University, which brought remarkable results for the ImageNet Challenge.
- They experiment with 6 models, with different numbers of trainable layers. Based on the number of models the two most popular models are VGG16 and VGG19.
Why and what of the pre-trained model?
- These are models, which are networks with a large number of parameters ( A Case in point is VGG16, which has 138 Million Parameters)
- Generally, training such a network is time and resource-consuming
- The pre-trained models for CV mostly are pretty general-purpose too
- We can use directly use these models if we pick up any of the 1000 classes it is trained with.
- Even if it’s a little bit different, we can remove the top layer and train the weight of that layer only (Transfer Learning)
What is this ImageNet dataset?
This was an initiative taken by Stanford Professor Fei-Fei Li in collaboration with wordnet from 2006. The image annotations were crowdsourced. This actually made the testbed of computer vision tasks really very robust, large, and expensive. Based on ImageNet a 1000 class classification challenge started with the name ImageNet Large Scale Visual Recognition Challenge (ILSVRC).
Actually, this competition is responsible for the birth of most of the prominent CNN models.
Importing Pretrained Model
Let’s take a look at the architecture for this pre-trained model
Now let’s implement our model.To make our work even easier we will use a sequential keras model and just two dense layer and one dropout layer.
Training Transfer Learning Model
Now we will proceed to train the model based on the VGG16 model and review the results of the last epochs.
We have get an accuracy of 92.58$ in training and 89.52% in validation dataset. Our learning curve shows a better performance at the beginningand using pre-trained model ,we get good values in accuracy at earlier epochs.
9. Final Scores
Finally we show the final results with the test dataset, obtaining an accuracy of 86.6%. We could make some more fine tunning to the model, but for now we will leave these values to generate some predictions.
10. Predictions
Let’s show some good predictions from transfer model.
Let’s show some bad predictions from transfer model.
Checking the erroneous predictions we can identify that misinterpretation of some classes. For example for actual fogsmog image, the model predicts that the image is snow. This prediction is logical since in training process the white color could be predominant to extract the main features in both cases. Similarly for the rain image, the predominant color is white, so the model fails to predicts this image.
For future works and improvements it will be important to do some feature map extraction to see how the CNN filters are getting relevant features.
11. Conclusions and Recommendations
In this article, we have established a database of wheater phenomena images under meteorological criterion. This database contains 3348 images with 6 wheater phenomena. We proposed a three CNN classification models using Keras+Tensorflow due to its simplicity and ease of code.
Two models have been implemented from scratch to find out their learning behavior with few samples in the dataset. Then a model has been implemented using transfer learning, more specifically using the VGG16 model where we get better perfomance on a pre-trained model.
From the results obtained these models can learn features of weather phenomena well, specially the one with transfer learning which is effective for weather classification. However, this models confuses some categories of weather phenomena, which may be due to the similarity and complexity of the images.
Overall, the classification accuracy of transfer model is as high as 90% for validation and 87% for testing. Therefore, the proposed model can be widely applied to the daily observation of weather phenomenon images and also can provide weather guidance for environmental monitoring, agriculture, and transportation, especially belonging to weather change and forecasting.
For future works, it would also be interesting to work on weather conditions and look for characteristics associated with one phenomena, with the aim of making better predictions. For example, we could consider building a model from the clouds types(cirrus, nimbus, stratus, etc.), these properties allow us to predict with a certain degree of certainty the presence or absence of precipitation (rain, hail, snow, storms, tornadoes, etc).
Github repository
https://github.com/urvog/weatherclassification
References
https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2020EA001604
