x days of code
Hi
Actually most of my daily routine is going with coding around data science / data engineering. So I
think that I can share some of them here.
This is a hard 'find the x' question. And I think I wont find the x soon.
I'll update how my process is going. You can check it.
-- X DAYS OF CODE --
UPDATE (2 MONTHS LATER)
This x=96 days of code challange was so good for me. Actually I didnt finish the x at 96 actually, I code everyday but writing about to process wasnt easy for me. School and moving one company to another was really an intense process for me. So ended my x-days-of-code challange.
I wanna write some benefits of this challange to me:
- Along these 96 days my motivation level was so high, so I boost myself to do more and learn more,
- After first couple weeks I realized doing smth contuniously teached me that I can do more, dive more,
- And end of these 96 days I took a job in one of the biggest e-commerce company (Hepsiburada) in Turkey (my country), Now I'm working with trillions/terebytes of data, this is an experience that so valuable for me.
- And 2 months later this challange ended I realized that I want it again to learn more and to do more.
I NEED BREAK A BREAK FOR COUPLE OF WEEKS. HOMEWORKS AND MIDTERM EXAMS SEASON IN SCHOOL ARE TAKING SO MUCH TIME!!!
I wanna write some benefits of this challange to me:
- Along these 96 days my motivation level was so high, so I boost myself to do more and learn more,
- After first couple weeks I realized doing smth contuniously teached me that I can do more, dive more,
- And end of these 96 days I took a job in one of the biggest e-commerce company (Hepsiburada) in Turkey (my country), Now I'm working with trillions/terebytes of data, this is an experience that so valuable for me.
- And 2 months later this challange ended I realized that I want it again to learn more and to do more.
I ENDED MY CHALLANGE BC IM SO BUSY WITH MY MOVING ONE COMPANY TO ANOTHER AND MASTER PROGRAM COURSES!!
Day 96 (17.11.2021)
I made my data structures and algorithms homework after working day with c. You can find it here as week 2.
Today There was one of the last lessons of my Data Engineering Bootcamp. We studied about an example with Spark and Kafka. I should build a project with these tools in 3 weeks. I learned some basics about these ones.
Today I got job offer as Data Analysis Specialist from "Hepsiburada" - ecommerce company that I mentioned #days52 to days 57-. In the new year, my level of knowledge will increase a lot. My learning process/trend continue to raise sharper.
And yea, today I code again. With my Data Science team, we did some fixes and committed our latest prediction of the test set. So there is nothing to do with project. 2 days later we'll present our process.
Friday I solved my Data Structures and Algorithms homework questions with c. Here is the codes. Actually I had no so much experience with c, and lecturer wanted this: "We should solve problems with no math functions. My codes work well.
We'll see.
Today we went a "developers meeting" to a bar with my friend. I meet more than 20 people there. There were awesome guyz :D
We started building model part of our Data Science Bootcamp Project. And we did a statistically succesfull approach about feature engineering parts.
- For observation_1 month_1 = January, month_2 = Jebruary maybe but, for observation_2 month_1 could be equal to March, and month_2 = April.
- So we decided to take difference of all sequential (with 12month lag) months, so we get 12 difference scores.
- And we calculated some statistics, and standardize them. This approach gave us some logical patterns.
- I liked our approach so much :D
Whole weekend I was with my Data Science Bootcamp team. We understood electricity data and decided how we can handle with the problems.
I have a lot of homework in my graduate classes. I did my homework these 2 days.
I'm in project week with my Data Science Bootcamp. We'll analyze Theft and Loss Estimation data about electricity.
Today I had data engineering bootcamp. We learned/repeated some about spark and hadoop.
And we talked about our projects, I'll built a project with twitter API.
Today's topic is technical interview about my project (from day 57) with a huge e-commerce company :D
I had 5 hours lesson today with Data Science Bootcamp. We studied about Time Series Analysis. Actually this area smth I knew, I refreshed my knowledge.
And I have a important technical meeting tomorrow. So I studied some theorical and applied parts to be ready. So I feel good :d
Along with this 2 days I focused 2 topic
- How can you handle with the classes that not included by training set? or Probability threshold of a classifier how could effect the result? etc.
- Actually there is some available solutions. But I handled with them again and It was informative experience to handle these in a hw.
I searched some about creating stream data APIs and I tried some. Actually creating API with web scraping looks not true way.
I searched some concepts and created a basic example of my next Web Scraping blog post.
And I did my homeworks of master/university lessons. (This was not directly about coding but the logic).
Today there was Data Engineering Bootcamp and we deep dive Spark and Kafka usage again.
Day Off. I visited my ex university (statistics department) and talked with my teachers. Today was so good.
And in the end of the day there was a meeting with my blog community (SistersLab).
Full day I had Data Science Bootcamp. We started to explore analysing steps, and which little touches we should do.
I studied again my old data engineering notes. And I updated my NiFi project. I added MongoDB writing processor.
I tried to make my own stream data API, but after that I realize this is not actually live stream, so I didnt add this to project but It was good things to try.
Day off, I dont count today. I created my good looking CV.
Today after working day there was additional lesson on my Data Science Bootcamp.
Today I had a course on Data Engineering Bootcamp. And we started to explore cloudera system deeply.
I'll watch this lesson again and study about it soon. There was so much details.

- So I sent data from Python to Kafka as producer
- Connedted Kafka with Nifi
- Kafka took data/messages and processed with Nifi
- And some conditions triggered Slack to send message
- Used JoltTransformJSON and the queries to select and transform data. (JoltTransformJSON is a good processor but little complicated for some tasks)
- Created some new fetures to trigger some events.
- Triggered Slack and ElasticSearch with the results. (I send message to Slack and wrote data to ElasticSearch)
- I connected nifi with Kafka.
- But I'm so fresh with Kafka, so I produced data from only bash of terminal (ubuntu). So there is no datasource yet.

Today I finished my project, later deployed it on cloud linux machine and watched logs to check are there any problem. And All done.
This week has been very tiring. Time to evaluate myself / process:
- Python, Flask/Dash, Docker, Linux
Sample pieces of the app:


And today is weekend, holiday? No. I had things to do
- Descriptive statistics task: 95% finished
- I should only add title for the plots, they will be dynamic but its ok, I can solve it less than 1 hour.
- Business report task: 100% finished
- Its ok. Np.
- Funnel Analysis: 50%
- I solved markov chain problem and implemented it to webb app.
- Tomorrow is the last day and the day that I handle with the funnel analysis.
Today after working day I continue to implement KPI task to web app. Visually everything is ok now.
- But Details are so much you know. There is some problems with the comment-plot couples.
- I took screenshots of the codes, Im showing them with the comments.
- But I dont keep them in folder/local. I upload them to github-issues, so github created a link for me. This is little tricky and awesome thing :D
- Descriptive statistics task: 70% finished
- There is so much detailed things to do. Actually I spend so much time with this but there is more details to do.
- All the tables created, comments and plots-dropdowns connections should fix, there is some errors.
- Business report task: 85% finished
- Screenshots of the codes added, most of the comments added.
- Results'll implement to dashboard/dash app
- Funnel Analysis: 30% finished
- Markov chain partly mostly solved, But this part should be added to web app, I didnt started this yet.
- I searched some notebooks about funnel analysis, but this task seems little hard for this week :/
Today after working day continueing was so tiring for me
- Descriptive statistics task: 50% finished
- summary tables'll be added
- for last 3 entity dynamic visuals'll be created
- Business report task: 65% finished
- Some repetitive tasks'll convert functions
- Results'll implement to dashboard/dash app
- Funnel Analysis: not started yet
- I read some articles about funnel analysis, so I have some insight about the task. I think I'll handle with this task well.
NOTE: I missed the service bus in the morning, I couldn't wake up :D
- I chose not to use SQL bc, I'll publish my app with docker on linux cloud machine. And I have some problem to create SQL on docker and manage it. Yea I know mysql could be good option on docker but as I said, this could make my process harder and this is a task that time consuming for me in this situation.
NOTE: Actaully I remember something like "funnel plot" from my graduation project of Univerty years. It is about Meta Analysis, and the scope of the data.
Actually today was not enough efficient for me.
Hey, I've been accepted to a data science bootcamp. I will be in a 6-week bootcamp, It is starting next week.
And today after work I took a 3 hours basic GoLang course. Tomorrow I'll start a GoLang bootcamp at the same time, this study class will take 12 days, I wanna be ready enough for this journey. And yeap, Go looks cool.
After work I decided to finish coding part of my final chapter of dataviz blog series. And Yeap, I finished it, except finilizing color tones of the cards :D This is a really hard task you know, I cant decide. I can share the steps I took when I create a dashboard with python:
- Locate cards on layout for plots,
- Fill the cars of plots by basic plots with target types,
- Add buttons and dropdown etc,
- Rebuilt plots with the real data,
- Relate and activate buttons/ dropdowns with plots,
- Make interface prettier and make codes cleaner,

I upped the gear in my deep learning process. During these days, I read the second part of Geron's book "Hands on ML". Actually I read the first part of older version of the book last year, and I liked it so much. But first edition of the book was about tensorflow 1, so I read early release of second edition with tensorflow 2.
I started to study on deep learning again. Actually I had notes from last year about CNNs, I checked them again and I took some courses. And I'm in a 4 days mini bootcamp(with a community) about deep learning. I checked some examples and apply them step by step to understand to idea behind them. Actually I really refreshed my knowledge again and my passion increased again.
I focused to tensorflow today. I followed some documantations and some github repos. Actually I found myself little weak about the preprocessing steps of image processing CC applications. But I made good progress today.
Today I did some practive with google cloud services. Actually I didnt use them deeply, only the thing I use are basics. But I focused to use them from terminal. The tools I practiced were:
I updated my twitter-earthquake-bot couple days ago. And I faced with some problems. These are:
(I added the photo afer 20 days bc there was no clear output bc of the threshold last 20 days)

12 Mostly Only thing that I do is taking Java course for last 10 days. Actually I remember that there was maybe 2 days taht I do nothing. Actually I like my process, I learned so much about the idea of basic concepts of the Java programming.


Today I hacked. Actually it's not purely hacking :d Data in the cloud mongodb database stolen. But
actually there was only scraped data about second hand cars. So no big deal, I increased security level
of my db.
Actually I'm kinda lucky bc My database has blown up in couple of weeks (so quickly). And on
my baby steps I didnt any big project so there was no any important data. If they didnt hacked me today,
Maybe later I could made big projects with no enough security. I'm feeling lucky (and dumpy :d)
I wrote 4th chapter of my blog on sisterslab. Actually there is no so much thing but I created a dashboard for this. You can find it here.
I'm taking Java course
nowadays. My aim is
not only taking some courses, I wanna start to use Java in my projects.
Till now I learned syntax and some basics, and now I'm trying to be better on OOP with java. The
process is going very good.
Actually today I did almost nothing. But I built my portfolio github-page and my xdaysofcode page (So current page :D).
During these 2 days I made a end-to-end project. Project and all the details are here.
Today I deployed my twitter-eartquake-bot application on cloud linux machine. Normally I'm using linux with interface but on cloud machine there is no interface, so deploying steps required some new codes to write for me. I learned usefull things today like:
Today I rent a cloud machine (linux) for 12 months. I had no so much experience with linux, I know some basics but I wanna do more. Today:
- Installed ubuntu GUI
- And I removed it after couple hours. Bc there was a problem with speed, it was not clear to use (center of my machine is Germany, distance could be problem)
Today I searched some Java apps. And I solved some entry level Java problems on Hackerrank. I'm at the begining of the Java. But I'm taking small steps. In short term maybe I wont be good at it but, in long term (maybe in 2 year) I know that I'll use it fluently.
Today I took a google cloud certification course.
I study about REST Api more today.
Today me and one of my friend checked my projects (smth like code review). And we decide that I can make my codes more efective.
- I redesigned some of my projects design to fit "Model Viev Controler".
- I eliminated repeated functions or lines. (for example)
Today I started to analyze the data I scraped online store. But after start to know data and analyze I realize that variation on independent variables are not enough. I was aim to predict Prices' of the products but prices are seem not continious variable to use regression. Im talking like this bc prizes are like 6499, 6999, 7499...
- There are about 3600 product but there is 18 different price labels.
- So this is smth like classification problem, and I dont find smth true to use regression or classification in this situation.
If I wanna have data engineering skills, I think I should improve my level of knowledge in REST API. And actually I'm having fun while handling with html procedures.
- I took some steps about how can I create a token in my APIs, and How can I verified users.
- I used Flask for APIs, and JWT.
After working day, I didn't do something important today.
- I attended to lesson of my course and we examined use cases of cassandra and some other NoSQL dbs.
- I solved little details around my data that I web scraped from online sales website. I'll start to analyze in couple days.
Today I study on MongoDB in the course I take. Actually I'm using mongoDB in office everyday but I'm
mostly using it on my apps to store images. In the course mostly we study on tabular data. It was good,
and enjoyful.
And today I solve some problem in an exam. There was an sql question in the exam, and it was smth
like this:
- This is good because I didnt face with a problem required 1 vs 2 Inner Join like this. And I didnt know that this is possible.
- But my solution was correct, it was smth like this:
SELECT TABLE2.name FROM TABLE2
INNER JOIN TABLE1 ON TABLE2.phone_number = TABLE1.caller OR TABLE2.phone_number = TABLE1.callee
GROUP BY TABLE2.name
HAVING sum(calls.total_time) <= 15
ORDER BY TABLE2.name
I solved some hackerrank problems and I read "Yapay Zeka Uygulamalari (4th edition)" from Prof. Dr. Cetin Elmas. Actually I bought thos book couple weeks ago, and today I have started to read it and apply. I refreshed my knowledges and I started to learn some mores.

I started to handle with the data I collected yesterday. None values and the data with no standard is a big problem. Today I worked on them about 2 hours. But there is more to do. None and nan are not acting same. I understood this so bad.
- cm, mm, or meters,
- screen resolutions 27'', 27' , 27 inch or 27INCH,
- freedos, dos or FreeDOS

In last days I spent my time to find a topic and a good website to do web scraping. And today i collected some of them from a online sales website. I wanna make a end to end data science project, so this one is first step. I collected data about phones, laptops, all in one pcs' and tablets. Now I have a real world dataset with more than 5k rows. Now there is so much things to do like cleaning/ standardizing.
- Taking urls of whole pages
- Taking urls of whole products on each pages
- Taking features and values of all products
