IT-Cloud

How to do Data Replication from MySQL, PostgreSQL, SQL Server, Oracle database using real time change data capture (CDC) to Google BigQuery

When you are dealing with analytic project, you will definitely go through these steps:1. Deciding your Data Sources2. Extracting and ingesting your Data from Source to Target3. Storing your Data in storage/database4. Transforming your data based on your business needs5. Consuming your data via explorer/reports/dashboards6. Activating your data via API, Machine Learning, export, integration One …

How to do Data Replication from MySQL, PostgreSQL, SQL Server, Oracle database using real time change data capture (CDC) to Google BigQuery Read More »

MyTip: Flowchart on how to choose the right model for your structured data ML use case

When talking about machine learning, besides the data, the model, and the prediction. There are few algorithm to choose to create the model whether it is popular such as linear regression, decision trees, even neural network. Or other algorithms such as k-Means, k-NN, SVM, Random Forests, Naive Bayes, logistic regression. This is the trick to …

MyTip: Flowchart on how to choose the right model for your structured data ML use case Read More »

MyLab: Database replication to BigQuery using change data capture

Databases like MySQL, Oracle, and SAP are the most often discussed CDC data sources. However, any system can be considered a data source if it captures and provides changes to data elements that are identified by primary keys. If a system doesn’t provide a built-in CDC process, such as a transaction log, you can deploy …

MyLab: Database replication to BigQuery using change data capture Read More »

MyLab: The best way to learn is to make your hands dirty! (RECAP)

I suggest you to run these labs to understand more about Data Engineering via Google Cloud. I still believe the best way to learn is to make your hands dirty! Please open https://www.cloudskillsboost.google/ and try these labs! BEST LABS FOR DATA ENGINEERING: Lab: Exploring a Public BigQuery Dataset  Lab: Analyzing Billing Data with BigQuery. Lab: …

MyLab: The best way to learn is to make your hands dirty! (RECAP) Read More »

MyTool: RVTools to analyze VMware vSphere environment

RVTools is a Windows .NET 4.6.1 application which uses VMware vSphere Management SDK 7.0 and CIS REST API to display information about your virtual environments. Interacting with VirtualCenter 4.x, ESX Server 4.x, VirtualCenter 5.x, ESX Server 5.x, VirtualCenter 6.x, ESX Server 6.x, VirtualCenter 7.0 and ESX server 7.0 RVTools is able to list information about …

MyTool: RVTools to analyze VMware vSphere environment Read More »

MyRef: Secure a BigQuery data warehouse that stores confidential data 

Many organizations deploy data warehouses that store confidential information so that they can analyze the data for a variety of business purposes. This document is intended for data engineers and security administrators who deploy and secure data warehouses using BigQuery. It’s part of a security blueprint that’s made up of the following: A GitHub repository that contains …

MyRef: Secure a BigQuery data warehouse that stores confidential data  Read More »

MyLab: Multiple VPC Networks

In this lab you create several VPC networks and VM instances and test connectivity across networks. Specifically, you create two custom mode networks (managementnet and privatenet) with firewall rules and VM instances as shown in this network diagram: The mynetwork network with its firewall rules and two VM instances (mynet-eu-vm and mynet-us-vm) have already been created for you in this Qwiklabs …

MyLab: Multiple VPC Networks Read More »

Visualize spend over time with Google Data Studio

Dashboard Sample: https://gcpbilling.page.link/dash You can get up-to-date Cloud Billing graphs throughout the day, and use labels to slice and dice your Google Cloud bill the way you want by combining Cloud Billing data export to BigQuery functionality with Google Data Studio.Step by Step: https://cloud.google.com/billing/docs/how-to/visualize-data You can use labels to answer complex questions, such as “How can I …

Visualize spend over time with Google Data Studio Read More »

Google Cloud setup checklist (Enterprise Guide)

This checklist helps you set up Google Cloud for scalable, production-ready enterprise workloads. The checklist is designed for administrators who are trusted with complete control over the company’s Google Cloud resources. Checklist Cloud identity and organization Users and groups Administrative access Set up billing Resource hierarchy Create initial projects in the resource hierarchy Confirm projects …

Google Cloud setup checklist (Enterprise Guide) Read More »

How to secure a BigQuery data warehouse that stores confidential data

This document is intended for data engineers and security administrators who deploy and secure data warehouses using BigQuery. It’s part of a security blueprint that’s made up of the following: A GitHub repository that contains a set of Terraform configurations and scripts. The Terraform configuration sets up an environment in Google Cloud that supports a data warehouse …

How to secure a BigQuery data warehouse that stores confidential data Read More »