List: Data Engineer Things | Curated by Thin Nguyen Duy

May 19, 2022

25 stories

Data Engineer Things
In
TDS Archive
by
Hadi Fadlallah
Installing Hadoop 3.1.0 multi-node cluster on Ubuntu 16.04 Step by StepThere are many links on the web about install Hadoop 3, many of them are not working well or need improvements. This article is taken from…
Nov 5, 2018
301
6
Nov 5, 2018
301
6
João Torres
How To Set Up a Hadoop 3.2.1 Multi-Node Cluster on Ubuntu 18.04 (2 Nodes)To start: What is Hadoop?
Feb 3, 2020
222
19
Feb 3, 2020
222
19
karcuta
Apache Spark Fundamentals — Phần 4: Spark Catalyst Optimizer1. Spark Catalyst Optimizer
Jun 8, 2020
113
Jun 8, 2020
113
In
The Slalom Blog
by
Amarpreet Singh
Introduction to Schema Registry in KafkaApache Kafka has been gaining so much popularity these past few years due to its highly scalable, robust, and fault-tolerant…
Apr 15, 2020
353
3
Apr 15, 2020
353
3
In
Geek Culture
by
Nirav Bhatt
Why Every Developer Should Learn CloudEasiest way to stay relevant in ever-changing market
Jul 6, 2021
8
Jul 6, 2021
8
In
TDS Archive
by
Antony Henao
Building a Data Warehouse: Basic Architectural principlesAn approach towards simplifying data warehousing
Feb 10, 2020
669
3
Feb 10, 2020
669
3
Lewis Gavin
How to architect the perfect Data WarehouseKey warehousing techniques made simple
Jul 26, 2018
1.8K
12
Jul 26, 2018
1.8K
12
In
SoftwareMill Tech Blog
by
Andrzej Ludwikowski
The best serialization strategy for event sourcingA brief summary of the existing serialization solutions (not only) for events in Event Sourced systems.
Jun 19, 2019
428
6
Jun 19, 2019
428
6
In
WhatfixEngineeringBlog
by
Saravanan A R
Real-Time data processing architecture using Apache Spark, Apache Kafka, and ClickHouseThis blog will talk about Real-time data processing architecture using Apache Spark, Apache Kafka, and ClickHouse.
Jan 3, 2022
29
1
Jan 3, 2022
29
1
In
bakdata
by
Victor Künstler
Continuous NLP Pipelines with Python, Java, and Apache KafkaAdvancements in machine learning, data analytics, and IoT, and the business strategic shift towards real-time data-driven decision making…
Jul 6, 2020
87
Jul 6, 2020
87
In
Oceanize Lab Geeks
by
Faysal Ahmed
Concepts of Database ArchitectureWhat Is Database Architecture?
Aug 31, 2017
514
4
Aug 31, 2017
514
4
In
ZaloPay Engineering
by
AJ Pham
Tìm hiểu bài toán đồng thuận trong hệ thống phân tán (Phần 1)Khái niệm đồng thuận có nghĩa là nhiều node cùng đồng ý/hủy bỏ về một cái gì đó — nó có thể là một giá trị, một quá trình hành động hoặc…
Mar 2, 2019
81
Mar 2, 2019
81
In
ZaloPay Engineering
by
Alex Nguyen
SQL Planning: Parser & OptimizerBài viết giới thiệu về cách SQL Engine xử lý câu truy vấn như thế nào
Aug 13, 2019
195
1
Aug 13, 2019
195
1
In
Robinhood
by
Arpan Shah
Taming ELKThe ELK (Elasticsearch[ES], Logstash and Kibana) stack is an extremely powerful and useful tool for powering search and analytics. At large…
May 3, 2017
155
May 3, 2017
155
In
Robinhood
by
Vineet Goel
Adding Faust to your Existing ArchitectureQuick Links: GitHub | Documentation
Aug 22, 2018
474
4
Aug 22, 2018
474
4
In
TDS Archive
by
Kiruparan Balachandran
Connecting the Dots (Python, Spark, and Kafka)Python, Spark, and Kafka are vital frameworks in data scientists’ day to day activities. It is essential to enable them to integrate these…
Jul 8, 2019
177
1
Jul 8, 2019
177
1
In
Insight
by
Sriram Baskaran
Airflow 101: Start automating your batch workflows with easeIn this blog, I cover the main concepts behind pipeline automation with Airflow and go through the code (and a few gotchas) to create your…
Oct 17, 2018
2.2K
5
Oct 17, 2018
2.2K
5
In
TDS Archive
by
Mustavi Sarker
Using Kafka to optimize data flow of your Twitter StreamA guide to processing big data from Twitter efficiently
Jul 30, 2020
99
1
Jul 30, 2020
99
1
 This story is no longer available
Adnan Siddiqi
Create your first ETL Pipeline in Apache Spark and PythonIn this post, I am going to discuss Apache Spark and how you can create simple but robust ETL pipelines in it. You will learn how Spark…
Jun 9, 2019
809
6
Jun 9, 2019
809
6

Data Engineer Things

Installing Hadoop 3.1.0 multi-node cluster on Ubuntu 16.04 Step by Step

There are many links on the web about install Hadoop 3, many of them are not working well or need improvements. This article is taken from…

How To Set Up a Hadoop 3.2.1 Multi-Node Cluster on Ubuntu 18.04 (2 Nodes)

To start: What is Hadoop?

Apache Spark Fundamentals — Phần 4: Spark Catalyst Optimizer

1. Spark Catalyst Optimizer

Introduction to Schema Registry in Kafka

Apache Kafka has been gaining so much popularity these past few years due to its highly scalable, robust, and fault-tolerant…

Why Every Developer Should Learn Cloud

Easiest way to stay relevant in ever-changing market

Building a Data Warehouse: Basic Architectural principles

An approach towards simplifying data warehousing

How to architect the perfect Data Warehouse

Key warehousing techniques made simple

The best serialization strategy for event sourcing

A brief summary of the existing serialization solutions (not only) for events in Event Sourced systems.

Real-Time data processing architecture using Apache Spark, Apache Kafka, and ClickHouse

This blog will talk about Real-time data processing architecture using Apache Spark, Apache Kafka, and ClickHouse.

Continuous NLP Pipelines with Python, Java, and Apache Kafka

Advancements in machine learning, data analytics, and IoT, and the business strategic shift towards real-time data-driven decision making…

Concepts of Database Architecture

What Is Database Architecture?

Tìm hiểu bài toán đồng thuận trong hệ thống phân tán (Phần 1)

Khái niệm đồng thuận có nghĩa là nhiều node cùng đồng ý/hủy bỏ về một cái gì đó — nó có thể là một giá trị, một quá trình hành động hoặc…

SQL Planning: Parser & Optimizer

Bài viết giới thiệu về cách SQL Engine xử lý câu truy vấn như thế nào

Taming ELK

The ELK (Elasticsearch[ES], Logstash and Kibana) stack is an extremely powerful and useful tool for powering search and analytics. At large…

Adding Faust to your Existing Architecture

Quick Links: GitHub | Documentation

Connecting the Dots (Python, Spark, and Kafka)

Python, Spark, and Kafka are vital frameworks in data scientists’ day to day activities. It is essential to enable them to integrate these…

Airflow 101: Start automating your batch workflows with ease

In this blog, I cover the main concepts behind pipeline automation with Airflow and go through the code (and a few gotchas) to create your…

Using Kafka to optimize data flow of your Twitter Stream

A guide to processing big data from Twitter efficiently

Create your first ETL Pipeline in Apache Spark and Python

In this post, I am going to discuss Apache Spark and how you can create simple but robust ETL pipelines in it. You will learn how Spark…

Thin Nguyen Duy

Reading list

Better Life

Mindfulness