Durga Prasad

Choudhury Durga Prasad

Senior Data Engineer | Lead Analytics Data Engineer | Cloud Agnostic

Explore My Work ↓

About Me

Results-oriented Data Engineer with over 5+ years of experience delivering scalable, cloud-native data solutions across AWS, Microsoft Azure, and Google Cloud Platform (GCP). Certified in all three platforms, I specialize in PySpark, DBT, Snowflake, Python, and SQL to design, build, and optimize data pipelines that drive real-time analytics and data-driven decisions.

I excel in end-to-end pipeline development, from data ingestion and transformation to validation and optimization, ensuring data quality, performance, and scalability. With strong data modeling expertise and a focus on business impact, I am committed to building efficient, high-performance data architectures.

With a proven track record of managing complex workflows and enabling actionable insights, I help organizations leverage data for smarter, faster decision-making.

4+

Years Experience

10+

Projects Completed

7+

Certifications

Working

Technical Skills

Programming Languages

Python 95%
PySpark 90%
SQL 90%
Scala 50%

Data Science & Analysis

NumPy/pandas 90%
Matplotlib/Seaborn 85%
Sigma Computing 80%
Tableau 80%

Data Engineering

Snowflake 90%
BigQuery 85%
Reactor UI/DSL 85%
Reactor Data 80%

Cloud Platforms

Google Cloud (GCP) 85%
Azure 80%
AWS 75%

Containerization & Orchestration

Docker 80%
Kubernetes 70%
Apache Airflow 85%

Data Modeling & Optimization

Data Modeling 90%
Query Optimization 85%
Data Integration 85%
Data Build Tool (DBT) 90%

Statistics & Fundamentals

Statistics 80%
Algorithms/Data Structures 75%
Version Control (Git) 85%

Professional Experience

Fabletics, LLC

Contract · Remote

Analytics Engineer – Blue Yonder Integrations

Nov 2025 – Jan 2026 · 3 mos
  • Led post go-live analytics engineering for Blue Yonder supply-chain integrations, stabilizing production data pipelines and improving trust in demand and inventory reporting.
  • Designed and optimized Snowflake data models for Merch Planning and Demand Planning, enabling faster analytics and more reliable planning insights.
  • Built and maintained Airflow-orchestrated ETL pipelines, ensuring consistent and timely data delivery across planning dashboards.
  • Partnered with Blue Yonder engineers, internal platform teams, and business stakeholders across time zones to translate planning requirements into scalable data solutions.
  • Diagnosed and resolved data quality and performance issues in production, reducing operational noise and manual intervention.
  • Used Codex and Cursor to accelerate development of Snowflake data models and Airflow pipelines, reducing implementation time while maintaining production-quality standards.
  • Leveraged AI-assisted code generation and refactoring to iterate quickly on complex Blue Yonder integration logic, allowing greater focus on data validation, performance tuning, and stakeholder feedback.
  • Implemented monitoring and alerting to proactively detect pipeline failures, improving system reliability in a remote, on-call environment.
  • Applied AI tools to debug SQL and pipeline issues faster in production, shortening turnaround time for fixes and improving overall delivery speed.

Softborne Technology Solutions Pte. Ltd.

Full-time · On-site

Analytical Data Engineer II

Sep 2023 – Sep 2025 · 2 yr · Chennai, Tamil Nadu, India
  • Designed and scaled cloud-native data pipelines using GCP (BigQuery, Dataflow, Pub/Sub, and Cloud Composer) to support near real-time analytics for global clients.
  • Built and optimized Snowflake and BigQuery data models for large retail and marketing datasets, improving query performance and cost efficiency.
  • Developed modular transformation logic using DBT, enabling faster onboarding of new data sources and easier long-term maintenance.
  • Implemented large-scale data processing using Python, Apache Beam, and Spark, ensuring stable performance as data volumes grew.
  • Built and deployed custom REST API connectors using Airbyte, running on Cloud Functions and Cloud Run for scalable and reliable data ingestion.
  • Tuned complex SQL queries and warehouse configurations to handle high-volume workloads while keeping compute costs under control.
  • Performed end-to-end data validation and reconciliation with source systems, consistently maintaining 99%+ data accuracy.
  • Used Gemini as a daily development assistant to accelerate Python, SQL, and DBT development, enabling faster feature delivery without compromising reliability.
  • Leveraged AI-generated suggestions to refactor transformations, optimize queries, and explore edge cases, improving code quality and long-term maintainability.
  • Applied AI support during pipeline debugging and validation, reducing investigation time and allowing greater focus on data accuracy and system design.
  • Collaborated remotely with product managers, analysts, and engineers to deliver data solutions aligned with business outcomes.

KPI Partners

Full-time · Hybrid

Data Engineer I

Jul 2022 – Jul 2023 · 1 yr · Hyderabad, Telangana, India
  • Built automated data pipelines using Apache Airflow, Python, and AWS, improving data availability and reducing manual reporting effort.
  • Designed scalable data models across BigQuery, Snowflake, and DynamoDB, supporting multi-user analytics and growing data volumes.
  • Developed reliable ETL workflows to process thousands of records per run with consistent performance.
  • Implemented data quality checks and monitoring to detect issues early and protect downstream analytics.
  • Worked closely with cross-functional teams to align data pipelines with reporting requirements and business needs.

DXC Technology

Full-time · Remote

Associate Professional Software Engineer

Aug 2021 - Jun 2022 · 11 mos · Bengaluru, Karnataka, India
  • Developed and maintained data pipelines using Azure Data Factory and Databricks, supporting enterprise-scale analytics workloads.
  • Optimized analytical storage using Azure Synapse Analytics and Delta Lake, improving query performance and reporting reliability.
  • Implemented secure data access using Azure Key Vault and role-based access control (RBAC), meeting enterprise security and compliance requirements.
  • Monitored and resolved pipeline issues using Azure Monitor and Log Analytics, reducing downtime and improving operational stability.
  • Tuned SQL queries and Spark jobs to ensure efficient execution on large datasets.

Cyber Crime Cell, Gurugram Police

Internship · Remote

Cyber Security Intern

Jun 2021 - Jul 2021 · 2 mos · Gurugram, Haryana, India
  • Provided first-level compliance monitoring and investigations.
  • Assisted with forensics analysis and fact gathering.
  • Assisted with vulnerability assessments and penetration testing for specific applications, services, networks, and servers as required.

Google

Internship · Remote

Google Cloud Ready Facilitator

Apr 2021 - Jun 2021 · 3 mos · Bengaluru, Karnataka, India
  • Learned Computing, Application Development, Big Data & Machine Learning using Google Cloud's training platform Qwiklabs.
  • Completed self-paced labs with temporary credentials to Google Cloud Platform for hands-on learning.

DXC Technology

Internship · Remote

Project Trainee

Feb 2021 - Apr 2021 · 3 mos · Bengaluru, Karnataka, India
  • The project aims to analyze and identify the data quality issues and strategies to mitigate these Issues in the given data. We need to start with a PowerPoint presentation that outlines the approach that we will be taking.
  • The client has agreed on a 3-week scope with the following 3 phases as follows - Data Exploration: Model Development
  • Created PowerPoint presentations outlining approaches for data quality improvement.

Featured Projects

Shopping Sales Project

Developed an end-to-end data pipeline using AWS services and Airflow for reporting from Google Sheets API data.

AWS Airflow Python ETL

Real-time Data Streaming

Implemented a real-time data ingestion system with Spark Streaming for processing high-velocity data.

Spark Kafka Scala Streaming

Flight Delay Prediction

Developed a machine learning model for predicting flight delays with 85% accuracy using historical data.

Python ML Pandas Scikit-learn

Customer Segmentation

Utilized R and clustering techniques to segment customers for targeted marketing campaigns.

R Clustering Data Analysis

Certifications

Google Cloud

Google Cloud Certified Professional Data Engineer

Microsoft Azure

Microsoft Certified: Fabric Analytics Engineer Associate

Microsoft Azure

Microsoft Certified: Azure Data Engineer Associate

AWS

AWS Certified Data Engineer - Associate

Microsoft Azure

Microsoft Certified: DevOps Engineer Expert

Microsoft Azure

Microsoft Certified: Azure Developer Associate

Microsoft Azure

Microsoft Certified: Azure Administrator Associate

AWS

AWS Certified Cloud Practitioner

My Resume

Get In Touch

durgaprasadchowdury@gmail.com

+91 9502192674

Pathapatnam, Andhra Pradesh, India

github.com/durgaprasadpr718
linkedin.com/in/durga-prasad-choudhury-75022314b