I've been into analytics like for ever. I began analytics while in school when I started playing around with Matlab. My colleagues and lecturers started giving me their research work to analyze their data. My interest into data science got spiked after I adopted ANN to model the start-up phase of a gast turbine for my masters project using data from Afam power plant.

I began making meagre earnings for my self from students and lecturers who often gave me their project data to work on and analyze. Mind you, prior to this I was actually teaching and earned about 20,000 naira as monthly salary. Afterwards, I realized teaching in secondary school was a complete waste of my time.

I decided to pursue analytics fulltime and I must confess that the skill puts bread on my table.
Although, I'm yet to get a full time data analytics role I have worked on projects I would never have dreamt of working on and lots of opportunities have been coming my way

I have always wanted to become a data engineering expert because, I have been nursing the dream of establishing an enterprise analytics department for the government and developing a real time Plant condition monitoring tool. With time more ideas of projects/strat-ups would pop up in my head 😌😌

The decision to take the leap into data engineering came about when one of the clients I worked with in the past asked me if I could automate the movement of data from Azure active directory to snowflake and then to Tableau. During this period I
resigned from working with a colleague of mine whom we often do data science projects mostly for foreign Nigerian students. Doing the student projects was really time consuming as it was difficult balancing combining both. I even ran into problems trying to do it as the quality of the assignment I did for three students was really poor(6 assignments in total). This was because my attention span was divided

Connecting Snowflake to Tableau with the aid of an ODBC driver was hassle free but I couldnt figure out how to create a pipeline to connect Azure data logs to Snowflake

The time frame of the project my client asked me if I could do elapsed but this was what drew me in to taking DE serious.

Due to the restriction on Nigerian debit cards, I couldn't use it to secure the 200 dollar Azure free credit and my cousin residing abroad tried to help but pulled out after he became suspicious when I asked him to use his phone number to complete the Azure free trial registration process.

One thing led to the other and the client that previously reached out to me stepped in to support me with Azure standard monthly subscription of 100 dollars

Let's work and build gpt for nairaland with LLM


You'll be in charge of the data scraping

DyingFetus:
Let's work and build gpt for nairaland with LLM


You'll be in charge of the data scraping

Let's connect on WhatsApp
Zero eight 1 three six three six five 6 zero three

Nice one OP.Following

My client tasked me to familiarized myself with Azure and data governance using Azure Purview and I was required to create a data map in Azure Purview that maps azure SQL database(cloud) and SQL server database on premise with the default Adventure Works database.
Note this assignment was not for a project but with the intent to get acquainted with data governance.

Doing the task was really daunting because it was my first experience working on the cloud 😁
Well... I was still able to create a data map that automatically classify's columns using regex and Microsoft built in default classification techniques.

Still on still, assigning sensitivity labels was still another subject matter that was hard... to grind. I struggled to understand how to assign sensitivity labels to the scanned data because creating sensitivity labels was done on the Microsoft Information Protection Portal. Each time I scanned my database both on premise and on the cloud the data map was unable unable to pick it up.

I figured if I could create a database having sensitive data like social security number (SSN) the sensitivity labels I created in MIP Portal would be picked up in the data map.

So I asked chatgpt to generate a ficitiotius table having SSN and other details. I then used the fictitious table to create a database in SQL server. Viola!! After scanning in Azure Purview data map the SSN was flagged as a sensitive data.

Azure Purview can be used for data lineage because it automatically captures the lineage flow of data through different sources like data lakes and databases. For this reason, I wanted to work on writing an article on how I was able to do the task with data lineage inclusive but my subscription and the use of Azure Purview ALONE had consumed about 113 dollars.

I adviced myself to stopeet!!!😭

Well done OP. I'm following.
I'm also interested in data engineering. But I'm currently just doing some web scraping.