Happy New Year, Nairalanders!

As we step into a brand new year, it's the perfect time to look back at the rich mix of stories that colored this great community in 2023. Through a meticulous effort on text mining and data analysis techniques, I have uncovered key insights on how these stories have mirrored significant national events as well as what they say about the collective Nigerian psyche.

I am therefore very excited to share with you my findings from a whole year's journey through Nairaland's frontpage. This isn't just about numbers and charts; it's about understanding what got us talking, debating, and sometimes agreeing. We'll see how our conversations reflected the bigger picture of what's happening in Nigeria and beyond.

This will be a very long post and thus be divided into 3 main sections.
1. Exploratory Data Analysis: In this section, I'll be delving into the Nairaland frontpage articles from 2023 to understand their basic structure, trends, and patterns. This will involve looking at the frequency of articles over time, the views and comments garnered, the most common words or phrases, and other characteristics. The goal is to get a general feel for the data

2. Themes Classification: Here i'll apply topic modeling techniques to the Nairaland articles to identify common themes that emerge throughout the year. This will reveal the most discussed topics on the platform, how they evolve over time, and how they relate to one another. This section helps summarize and categorize the vast amount of textual content into coherent, understandable topics.

3. Sentiment Analysis: This section will focus on analyzing the sentiments expressed in the comments left by users on these frontpage articles. By examining over two million comments left across the 12 months of 2023, the aim will be to understand the general tone and opinion of the Nairaland community. Are the comments mostly positive, negative, or neutral? How do sentiments change with different types of articles or over time? This analysis will provide insights into the community's reactions and feelings toward various topics discussed in 2023. (THIS SECTION WILL BE ADDED AT A LATER TIME, SO BE SURE TO CHECK BACK)

Before i dive into the insights, here is a little about me. I'm Timothy Adegbola, a data scientist with a deep passion for sharing my knowledge and ideas. Currently based in Manchester, United Kingdom where I'm pursuing a master's degree in Artificial Intelligence. If you would love to connect with me, don't hesitate to check the info on my profile

Not sure under which section to write this post, I decided to put to use an Intelligent Text Classifier I recently built for Nairaland.com. The A.I model decided that GENERAL is the most plausible section to use

EXPLORATORY DATA ANALYSIS

1. POST COUNT

A total of 18881 news articles reached Nairaland front page in 2023

2. MONTHLY DISTRIBUTION OF ARTICLES

Total Number of Articles Published by Month
Jan: 2642 articles
Feb: 2175 articles
Mar: 1419 articles
Apr: 1518 articles
May: 1837 articles
Jun: 1587 articles
Jul: 1431 articles
Aug: 1103 articles
Sep: 1217 articles
Oct: 1153 articles
Nov: 1273 articles
Dec: 1526 articles

3. CATEGORY COUNT

There are 43 distinct categories of News/Articles on the Frontpage in 2023

4. CATEGORY DISTRIBUTION

Politics: 9854
Crime: 1080
Celebrities: 815
Sports: 676
Travel: 604
Romance: 601
Family: 597
Business: 565
Foreign Affairs: 533
Education: 465
Religion: 376
Health: 345
Jobs/Vacancies: 280
Food: 264
Career: 250
TV/Movies: 211
European Football (EPL, UEFA, La Liga): 183
Phones: 153
Properties: 128
Music/Radio: 127
Islam for Muslims: 122
Car Talk: 105
General: 95
Culture: 79
Agriculture: 69
Fashion: 53
Events: 49
Science/Technology: 36
NYSC: 35
Autos: 27
Investment: 26
Pets: 20
Webmasters: 13
Programming: 10
Forum Games: 8
Literature: 7
Art, Graphics & Video: 6
Computers: 5
Jokes Etc: 3
Gaming: 3
Technology Market: 1
Entertainment: 1
Business To Business: 1

5. TOP 10 CATEGORY (According to number of Published News Article)

Politics 9,854 Articles
Crime 1,080 Articles
Celebrities 815 Articles
Sports 676 Articles
Travel 604 Articles
Romance 601 Articles
Family 597 Articles
Business 565 Articles
Foreign Affairs 533 Articles
Education 465 Articles

6. BOTTOM 10 CATEGORY(According to number of Published News Article)

Programming 10 Articles
Forum Games 8 Articles
Literature 7 Articles
Art, Graphics & Video 6 Articles
Computers 5 Articles
Jokes Etc 3 Articles
Gaming 3 Articles
Technology Market 1 Articles
Entertainment 1 Articles
Business To Business 1 Articles

7. Total Number of Views

Total views of all the news articles on frontpage: 553,052,909

8. TOP 10 CATEGORIES (According to Total Views Garnered

Politics 249,551,840 Views
Crime 34,181,166 Views
Celebrities 32,816,489 Views
TV/Movies 30,834,726 Views
Romance 23,136,597 Views
Travel 18,913,254 Views
Family 18,432,661 Views
Sports 17,977,742 Views
Foreign Affairs 17,205,407 Views
Business 15,006,115 Views

9. BOTTOM 10 CATEGORIES (According to the total views garnered)

Art, Graphics & Video 274,927 Views
Programming 235,399 Views
Literature 215,001 Views
Computers 206,974 Views
Webmasters 194,213 Views
Jokes Etc 135,037 Views
Technology Market 53,765 Views
Gaming 50,285 Views
Business To Business 28,695 Views
Entertainment 19,997 View

10. TOP 10 CATEGORY (According to the highest average views

TV/Movies...............................146,136 Views Per frontpage post
Technology Market...............53,765 Views Per frontpage post
Pets.........................................47,019 Views Per frontpage post
Art, Graphics & Video........... 45,821 Views Per frontpage post
Jokes Etc.................................45,012 Views Per frontpage post
Fashion...................................44,114 Views Per frontpage post
Forum Games........................42,903 Views Per frontpage post
Computers.............................41,395 Views Per frontpage post
Celebrities...............................40,266 Views Per frontpage post
Food.........................................39,248 Views Per frontpage post

11. BOTTOM 10 CATEGORY (According to the highest average views per post)


Politics...............................25,325 Views per Frontpage Post
Career...............................24,248 Views per Frontpage Post
Programming...............................23,540 Views per Frontpage Post
Music/Radio...............................22,919 Views per Frontpage Post
Entertainment...............................19,997 Views per Frontpage Post
European Football (EPL, UEFA, La Liga)...............................19,492 Views per Frontpage Post
Jobs/Vacancies...............................18,642 Views per Frontpage Post
Gaming...............................16,762 Views per Frontpage Post
Webmasters...............................14,939 Views per Frontpage Post
Islam for Muslims...............................7,763 Views per Frontpage

12. Article with highest Views: Category by Category

This analysis focuses on the individual frontpage post with the highest views for all the 43 Categories represented on the frontpage of this forum in 2023. For instance

The most viewed post under POLITICS was a post titled 2023 Presidential Election Results From Polling Units, it garnered
444494 views, 441 comments and was published on 2023-02-25

The most viewed post under CAREER was a post titled I Earn £50 As A Naked Cleaner, it garnered
99622 views, 163 comments and was published on 2023-03-14

The complete list is shown below

You tried

13. TRENDS OF AVERAGE VIEWS ACROSS THE MONTHS

The line graph displays the trend of average views per month for Nairaland frontpage articles throughout 2023. It appears that there was a significant peak in views around March, followed by a general decline until a slight uptick in June. After June, the average views exhibit a declining trend, with a steep drop after September, reaching the lowest point in December. This pattern might suggest seasonal interest or events that drove higher readership at certain times of the year, with a noticeable drop-off towards the end of the year.

COMMENT ANALYSIS

14. TOTAL NUMBER OF COMMENTS

Total number of comments on all frontpage articles of 2023: 2,162,654

15. OVERALL TOP 20 POSTS WITH HIGHEST NUMBER OF COMMENTS
The post with highest number of comments is about a confession of a lady who locked up her husband for up to two days posted on 22nd August 2023. It garnered almost a thousand comments

16. TOP 10 CATEGORIES WITH THE HIGHEST TOTAL COMMENTS

Politics 1,082,954 Comments
Crime 110,951 Comments
Family 100,789 Comments
Romance 94,944 Comments
Celebrities 92,161 Comments
Foreign Affairs 78,104 Comments
Travel 73,351 Comments
Business 65,894 Comments
Sports 63,685 Comments
Education 54,627 Comments

17. BOTTOM 10 CATEGORIES WITH THE LEAST TOTAL COMMENTS

Webmasters 1169
Programming 1119
Literature 988
Computers 831
Art, Graphics & Video 669
Jokes Etc 510
Gaming 300
Technology Market 297
Business To Business 85
Entertainment 59

I am a bit disappointed that WEBMASTERS, PROGRAMMING, COMPUTERS has such a low number of discussion. It generally shows that most of the audience here are not tech savvy

18. POSTS WITH HIGHEST NUMBER OF COMMENTS: CATEGORY BY CATEGORY

Here i wrote a script that group the dataframe containing the articles by the 'Category' column, then wrote a lambda function that sorts the article in descending order by 'Comment Count' within each category group.
The result is displayed below

19. COMMENT DISTRIBUTION HISTOGRAM
The distribution is heavily right skewed which indicate that majority of the posts have less than 200 comments(with peak around 90 to 100 comments) and very very few have over 500 comments and it is exceptionally rare to have a post with over 800 comments.

20. Distribution of Posts by Hour of the Day

This shows what time of the day post are pushed to the frontpage. From the visualization below, it is clear that the mods favors evening around 8pm and 10 pm to send the posts. Further analysis reveals that these post are the most engaging in terms of comments and views. Very little post are published between 12am to 4am

20.PERCENTAGE OF POST WITH/WITHOUT SOURCE URLS

To do this i have to use regex to extract url from all the articles body and add to a separate column, then use a python function isna() to filter the rows with missing source url which I found out to be 27.1% as against 72.9% with at least 1 source url

21. TOP CITED DOMAINS

www.youtube.com 2582
twitter.com 1519
punchng.com 1255
www.vanguardngr.com 741
dailytrust.com 605
dailypost.ng 479
m.facebook.com 371
www.google.com 357
www.thecable.ng 348
igberetvnews.com 317
www.instagram.com 297
saharareporters.com 296
leadership.ng 284
.ng 272
www.premiumtimesng.com 238
www.channelstv.com 238
thenationonlineng.net 225
www.nairaland.com 221
www.facebook.com 184
www.lindaikejisblog.com 166

TO BE CONTINUED...

Vokallond1:
20.PERCENTAGE OF POST WITH/WITHOUT SOURCE URLS

To do this i have to use regex to extract url from all the articles body and add to a separate column, then use a python function isna() to filter the rows with missing source url which I found out to be 27.1% as against 72.9% with at least 1 source url

21. TOP CITED DOMAINS

www.youtube.com 2582
twitter.com 1519
punchng.com 1255
www.vanguardngr.com 741
dailytrust.com 605
dailypost.ng 479
m.facebook.com 371
www.google.com 357
www.thecable.ng 348
igberetvnews.com 317
www.instagram.com 297
saharareporters.com 296
leadership.ng 284
.ng 272
www.premiumtimesng.com 238
www.channelstv.com 238
thenationonlineng.net 225
www.nairaland.com 221
www.facebook.com 184
www.lindaikejisblog.com 166

Hello world! I am Regex.

22. PERCENTAGE OF POSTS WITH SOURCE URL (CATEGORY BY CATEGORY)

Crime 90.370370
Pets 85.000000
Politics 84.402273
Art, Graphics & Video 83.333333
Celebrities 83.312883
Investment 80.769231
Music/Radio 79.527559
Business 77.168142
Education 76.559140
Sports 73.964497
Literature 71.428571
Programming 70.000000
Foreign Affairs 69.043152
Autos 66.666667
Culture 65.822785
Travel 65.562914
Events 65.306122
Health 61.739130
Webmasters 61.538462
Science/Technology 61.111111
Properties 58.593750
TV/Movies 58.293839
NYSC 54.285714
Religion 54.255319
Fashion 47.169811
Phones 47.058824
Agriculture 46.376812
Car Talk 40.000000
Food 34.848485
Career 33.200000
Romance 31.281198
Islam for Muslims 28.688525
Family 28.643216
General 24.210526
Computers 20.000000
Jobs/Vacancies 14.285714
European Football (EPL, UEFA, La Liga) 6.010929
Business To Business NaN
Entertainment NaN
Forum Games NaN
Gaming NaN
Jokes Etc NaN
Technology Market NaN

N-GRAM ANALYSIS

N-grams are contiguous sequences of 'n' items from a given sample of text or speech. The 'items' can be phonemes, syllables, letters, words, or base pairs according to the application. The 'n' in n-grams represents the number of elements in the sequence. N-grams are used in various fields of linguistic analysis and text mining.

For example the phrase "My name is Timothy"
It has 4 unigrams i.e 'My', 'name', 'is', 'Timothy'.
It has 3 bigrams i.e 'My name', 'name is', 'is Timothy'
It has 2 trigrams i.e 'My name is', 'name is Timothy'

For this analysis i wrote a function that takes a DataFrame, a column name, an n-gram value (ngram_val), and the number of top n-grams to display (top_n). It uses CountVectorizer to count the frequency of n-grams in the specified column. It then displays the top n-grams and their frequencies in both a printed list and a horizontal bar chart.

WORDCLOUD VISUALIZATION

Word cloud visualizations, also known as tag clouds or text clouds, are graphical representations of word frequency that give greater prominence to words that appear more frequently in a source of text. The size of each word in the word cloud indicates its frequency or importance: larger words represent more commonly used words within the text, while smaller words correspond to less frequent terms. Below are the result of N-GRAM ANALYSIS and WORDCLOUD VISUALIZATIONS of all frontpage posts in 2023 under 3 sub-category

1. Article Titles
2. Article Bodies
3. User Comments

Unigrams of Article Titles

tinubu 1581
video 1035
obi 1000
photo 978
nigeria 870
nigerian 834
apc 780
state 771
peter 727
lagos 724
court 652
election 623
picture 619
president 591
2023 555
man 548
pdp 503
governor 478
buhari 442
naira 433

Bigrams of Article Titles

peter obi 695
president tinubu 173
supreme court 165
campaign rally 150
labour party 138
naira note 135
presidential campaign 132
president buhari 102
presidential election 99
apc presidential 91
bola tinubu 90
2023 election 88
subsidy removal 88
appeal court 88
new naira 77
presidential rally 74
governorship election 71
nigerian man 69
video photo 69
polling unit 69

Trigrams of Article Titles

presidential campaign rally 101
old naira note 58
apc presidential campaign 56
new naira note 54
big brother titan 41
peter obi campaign 39
picture peter obi 38
peter obi presidential 36
town hall meeting 35
apc presidential rally 35
peter obi datti 33
2023 live update 30
guinness world record 29
fuel subsidy removal 27
live update thread 26
evicted big brother 24
picture apc presidential 24
2023 presidential election 22
brother titan bbtitans 21
pdp presidential campaign 21

Unigram of Article Bodies

state 29127
nigeria 15940
government 13867
people 13688
president 13260
party 12261
nigerian 12115
governor 11801
election 10037
national 8904
court 8795
tinubu 8160
year 7820
country 7636
time 7592
candidate 6943
apc 6897
presidential 6781
2023 6544
mr 6494

Bigram of Article Bodies

federal government 2420
local government 2275
presidential candidate 2227
labour party 2215
bola tinubu 2214
lagos state 1999
progressive congress 1868
peter obi 1799
state government 1707
president bola 1661
democratic party 1649
state governor 1647
people democratic 1571
presidential election 1532
national assembly 1398
government area 1363
river state 1356
social medium 1335
supreme court 1298
congress apc 1282

Trigram of Article Bodies

people democratic party 1532
local government area 1359
progressive congress apc 1277
democratic party pdp 1012
president bola tinubu 1008
bola ahmed tinubu 946
independent national electoral 877
national electoral commission 851
central bank nigeria 831
president muhammadu buhari 748
electoral commission inec 592
state house assembly 552
state police command 498
labour party lp 470
federal high court 455
federal capital territory 445
federal republic nigeria 403
bank nigeria cbn 401
public relation officer 398
president bola ahmed 385

UNIGRAM OF ALL COMMENTS ON FRONTPAGE i.e most common word used in the comments

people 227537
dont 173424
know 172834
nigeria 153036
man 131353
tinubu 118342
make 110940
state 103316
obi 98949
want 96913
say 94646
time 94073
nigerian 93006
god 90989
thing 86140
come 85617
country 84751
money 81344
think 75388
good 74808

BIGRAM OF ALL COMMENTS ON FRONTPAGE

peter obi 27606
dont know 27426
dont want 9434
social medium 7481
lagos state 7003
god bless 6925
supreme court 6819
south east 6739
dont think 6473
dont understand 6140
presidential election 5914
labour party 5780
dont need 5498
igbo man 5212
people dont 4954
make sure 4480
river state 4357
make una 4321
year old 4224
jesus christ 4195

TRIGRAM OF ALL COMMENTS ON FRONTPAGE

bola ahmed tinubu 2265
vote peter obi 1163
president federal republic 1040
muslim muslim ticket 1035
federal republic nigeria 1009
people dont know 1002
asiwaju bola ahmed 934
new naira note 861
dont know people 809
free fair election 796
peter obi president 709
peter obi win 644
lagos state government 641
old naira note 613
lagos man land 570
president bola ahmed 556
win presidential election 552
god bless nigeria 527
long time ago 526
state house assembly 520

THEME CLASSIFICATION (Topic Modelling of News Article Title + Bodies)

Topic modeling is a type of statistical model for discovering the abstract "topics" that occur in a collection of documents. It is a powerful tool for summarizing large datasets of text by grouping and clustering words that frequently appear together. This technique helps to uncover hidden thematic structures within the texts, enabling us to see patterns that might not be apparent from a simple read-through. In essence, topic modeling can reveal the underlying themes or subjects in a body of text, making it easier to understand and categorize large volumes of textual information. I applied this technique to the entire frontpage posts in 2023 and set the number of clusters to 16. Below are the result and interpretation

Topic 0:
president: 0.0121
tinubu: 0.0103
nigeria: 0.0103
state: 0.0093
nigerian: 0.0086
people: 0.0076
election: 0.0069
presidential: 0.0067
party: 0.0065
country: 0.0063

Topic 0 revolves around political discussions, focusing on the presidency, elections, and national issues in Nigeria.


Topic 1:
police: 0.0172
state: 0.0114
suspect: 0.0072
arrested: 0.0070
officer: 0.0067
area: 0.0066
command: 0.0057
security: 0.0052
victim: 0.0052
incident: 0.0051

Topic 1 pertains to law enforcement and crime, with mentions of police activity, arrests, and security incidents.

Topic 2:
state: 0.0205
party: 0.0187
governor: 0.0186
apc: 0.0133
pdp: 0.0127
election: 0.0099
candidate: 0.0098
people: 0.0091
national: 0.0082
senate: 0.0078

Topic 2 appears to be about political party activities and governance, including elections and the workings of state government.

Topic 3:
wife: 0.0100
woman: 0.0097
man: 0.0090
god: 0.0085
marriage: 0.0085
husband: 0.0080
dont: 0.0074
child: 0.0073
love: 0.0069
married: 0.0069

Topic 3 relates to personal and social topics, possibly about relationships, religious beliefs, and family matters.

Topic 4:
government: 0.0124
nigeria: 0.0105
subsidy: 0.0099
price: 0.0096
per: 0.0088
fuel: 0.0079
oil: 0.0071
nigerian: 0.0070
president: 0.0067
state: 0.0066

Topic 4 discusses governmental policy, economic issues such as subsidies and fuel prices, and possibly national governance.


Topic 5:
court: 0.0239
election: 0.0200
inec: 0.0147
party: 0.0142
state: 0.0116
electoral: 0.0105
tribunal: 0.0099
petition: 0.0089
candidate: 0.0088
appeal: 0.0087

Topic 5 seems to deal with legal proceedings, possibly election-related court cases, and the judiciary system.

Topic 6:
rally: 0.0281
tinubu: 0.0269
obi: 0.0268
peter: 0.0253
peter obi: 0.0250
president: 0.0217
campaign: 0.0214
presidential: 0.0207
picture: 0.0190
state: 0.0185

Topic 6 is centered on political campaigns and rallies, particularly mentioning notable figures and presidential campaigns.

Topic 7:
job: 0.0103
health: 0.0066
year: 0.0062
work: 0.0061
nigeria: 0.0059
people: 0.0059
dont: 0.0057
time: 0.0053
get: 0.0051
company: 0.0050

Topic 7 touches on employment, health, and general life advice or discussions about societal issues.

Topic 8:
video: 0.0078
singer: 0.0076
music: 0.0070
wife: 0.0066
davido: 0.0062
mohbad: 0.0059
brother: 0.0056
housemate: 0.0055
death: 0.0053
baby: 0.0053

Topic 8 is about entertainment, focusing on music, celebrities, and possibly reality TV, It also touch on the painful demise of popular artiste MOHBAD

Topic 9:
bank: 0.0265
cbn: 0.0162
note: 0.0158
naira: 0.0149
old: 0.0105
nigeria: 0.0086
emefiele: 0.0083
court: 0.0083
new: 0.0082
policy: 0.0077

Topic 9 is concerned with financial topics, specifically banking, currency, and monetary policy in Nigeria.

Topic 10:
league: 0.0173
cup: 0.0136
world: 0.0132
player: 0.0125
club: 0.0117
goal: 0.0116
team: 0.0109
game: 0.0109
football: 0.0107
season: 0.0088

Topic 10 covers sports, likely focusing on football leagues, cups, and teams.

Topic 11:
university: 0.0174
student: 0.0164
school: 0.0112
nigeria: 0.0081
nigerian: 0.0075
education: 0.0074
state: 0.0070
year: 0.0060
also: 0.0060
institution: 0.0053

Topic 11 is centered on education, discussing universities, students, and educational policies.

Topic 12:
road: 0.0165
state: 0.0158
government: 0.0114
lagos: 0.0110
building: 0.0078
governor: 0.0075
project: 0.0071
land: 0.0061
also: 0.0060
bridge: 0.0060

Topic 12 focuses on infrastructure, urban development, and state-level governance, with mentions of roads and projects.

Topic 13:
airline: 0.0214
aircraft: 0.0159
air: 0.0155
airport: 0.0136
nigerian: 0.0115
flight: 0.0114
russian: 0.0105
russia: 0.0104
ukraine: 0.0102
aviation: 0.0100

Topic 13 might be about the aviation industry and related news, including international affairs that intersect with aviation.

Topic 14:
hamas: 0.0422
israel: 0.0406
gaza: 0.0376
israeli: 0.0365
palestinian: 0.0223
attack: 0.0137
killed: 0.0129
hostage: 0.0125
idf: 0.0121
war: 0.0110

Topic 14 discusses international conflict and geopolitics, specifically the Israeli-Palestinian conflict.

Topic 15:
big brother: 0.1194
big: 0.1029
brother: 0.0991
big brother titan: 0.0884
brother titan: 0.0882
titan: 0.0867
africa: 0.0729
show: 0.0536
host: 0.0384
channel: 0.0378

Topic 15 is focused on entertainment, likely related to a reality TV show, possibly "Big Brother Naija," given the keywords mentioned.

THEME CLASSIFICATION RESULT

A clustering algorithm was used to classify the frontpage post of 2023 into 16 classes, below were the results

SENTIMENT ANALYSIS (Coming Soon)...