Durga Prasad

Choudhury Durga Prasad

Senior Data Engineer | Lead Analytics Data Engineer | Cloud Agnostic

Explore My Work ↓

About Me

Results-oriented Data Engineer with over 5+ years of experience delivering scalable, cloud-native data solutions across AWS, Microsoft Azure, and Google Cloud Platform (GCP). Certified in all three platforms, I specialize in PySpark, DBT, Snowflake, Python, and SQL to design, build, and optimize data pipelines that drive real-time analytics and data-driven decisions.

I excel in end-to-end pipeline development, from data ingestion and transformation to validation and optimization, ensuring data quality, performance, and scalability. With strong data modeling expertise and a focus on business impact, I am committed to building efficient, high-performance data architectures.

With a proven track record of managing complex workflows and enabling actionable insights, I help organizations leverage data for smarter, faster decision-making.

4+

Years Experience

10+

Projects Completed

7+

Certifications

Working

Technical Skills

Programming Languages

Python 95%
PySpark 90%
SQL 90%
Scala 50%

Data Science & Analysis

NumPy/pandas 90%
Matplotlib/Seaborn 85%
Sigma Computing 80%
Tableau 80%

Data Engineering

Snowflake 90%
BigQuery 85%
Reactor UI/DSL 85%
Reactor Data 80%

Cloud Platforms

Google Cloud (GCP) 85%
Azure 80%
AWS 75%

Containerization & Orchestration

Docker 80%
Kubernetes 70%
Apache Airflow 85%

Data Modeling & Optimization

Data Modeling 90%
Query Optimization 85%
Data Integration 85%
Data Build Tool (DBT) 90%

Statistics & Fundamentals

Statistics 80%
Algorithms/Data Structures 75%
Version Control (Git) 85%

Ingestion & Integration

Fivetran 90%
dbt Core 90%
SnapLogic 85%
AWS Services (Glue, S3, AppFlow) 80%

Professional Experience

Cents

Contract · Remote

Senior Data Engineer

Nov 2025 – Present
  • Owned the company's core analytics data platform end to end on Snowflake and dbt, serving three product brands (Cents, Laundroworks, Starchup) and establishing it as the single source of truth for GMV, ARR, payments, and product analytics relied on by finance, sales, product, and the executive team.
  • Defined the warehouse architecture and modeling standards across staging, harmonized, and presentation layers in dbt, scaling the codebase to dozens of governed, tested models and replacing fragile Looker derived tables with a consistent, trusted semantic layer in Looker.
  • Led source analysis and integration across many third-party connectors spanning payments, CRM, finance, product, and accounting systems (Stripe, Salesforce, FullStory, Xero, and more), evaluating each provider's data model against business requirements and onboarding them into Snowflake to broaden analytics coverage.
  • Built product data engineering pipelines that model application event, behavioral, and usage data into analytics-ready datasets, powering product, funnel, and feature-adoption metrics for product and growth teams.
  • Productionized the enterprise payments analytics pipeline from ingestion through Snowflake task orchestration and dbt presentation views, delivering payment-method, top-up, and membership revenue reporting that unblocked a key enterprise customer rollout.
  • Orchestrated the platform end to end with Fivetran-managed syncs, GitHub Actions workflows, and AWS services (Glue, S3, AppFlow) coordinating 150+ Snowflake tasks, advancing daily data SLAs to before business hours while hardening reliability through schema contracts and cross-source reconciliation.
  • Drove an AI-assisted engineering workflow with Claude Code and Codex for model development, refactoring, and debugging, cutting delivery time on complex models while maintaining production standards through tested, peer-reviewed code.
  • Partnered with finance, sales, and leadership to define and document metric standards for GMV, ARR, and payments, increasing trust and adoption of self-serve analytics across the organization.

Collinson Group

Contract · Remote

Senior Analytics Engineer

Mar 2026 – Present
  • Engineered the analytics infrastructure for Partner 360, the enterprise data platform powering Collinson's global airport-lounge and Priority Pass business, with dbt on Snowflake orchestrated through dbt Cloud and continuous delivery via GitHub Actions.
  • Designed and delivered multiple analytics subdomains end to end, including consumer feedback, lounge visits, and flight-schedule forecasting, building layered dbt models from staging through fact and aggregate that feed partner-facing dashboards and the data science team.
  • Built the data foundation for the data science team's lounge-occupancy forecasting, modeling flight-schedule and visit data into 15-minute aggregates and eliminating a double-counting defect that had distorted demand signals feeding revenue forecasts.
  • Established data-quality and testing frameworks as reusable dbt macros with production-calibrated thresholds, embedding automated validation into every build so releases shipped with reviewable evidence and far fewer regressions.
  • Standardized cross-market analytics by versioning survey-question weighting and rating logic, making lounge satisfaction scoring comparable across regions and resolving systemic data-integrity issues such as stale joins and locale calendar defects that had undermined visit reporting.
  • Modeled Salesforce partnership, opportunity, product, and cost data into governed fact and aggregate layers, enabling partner profitability and product-cost analysis across the global lounge network.
  • Owned ingestion and transformation across SnapLogic into Snowflake, consolidating dbt as the single transformation and orchestration layer to simplify the platform and cut operational overhead.
  • Tuned Snowflake and dbt performance with incremental models and efficient joins over large flight-schedule and visit datasets, keeping build times and compute cost in check as data scaled.
  • Accelerated delivery with Codex across model authoring, macro development, and PR review, reducing turnaround on complex models and clearing multi-reviewer feedback faster while adhering to repo standards.

Fabletics, LLC

Contract · Remote

Analytics Engineer – Blue Yonder Integrations

Nov 2025 – Jan 2026 · 3 mos
  • Led post go-live analytics engineering for Blue Yonder supply-chain integrations, stabilizing production data pipelines and improving trust in demand and inventory reporting.
  • Designed and optimized Snowflake data models for Merch Planning and Demand Planning, enabling faster analytics and more reliable planning insights.
  • Built and maintained Airflow-orchestrated ETL pipelines, ensuring consistent and timely data delivery across planning dashboards.
  • Partnered with Blue Yonder engineers, internal platform teams, and business stakeholders across time zones to translate planning requirements into scalable data solutions.
  • Diagnosed and resolved data quality and performance issues in production, reducing operational noise and manual intervention.
  • Used Codex and Cursor to accelerate development of Snowflake data models and Airflow pipelines, reducing implementation time while maintaining production-quality standards.
  • Leveraged AI-assisted code generation and refactoring to iterate quickly on complex Blue Yonder integration logic, allowing greater focus on data validation, performance tuning, and stakeholder feedback.
  • Implemented monitoring and alerting to proactively detect pipeline failures, improving system reliability in a remote, on-call environment.
  • Applied AI tools to debug SQL and pipeline issues faster in production, shortening turnaround time for fixes and improving overall delivery speed.

Softborne Technology Solutions Pte. Ltd.

Full-time · On-site

Analytical Data Engineer II

Sep 2023 – Sep 2025 · 2 yr · Chennai, Tamil Nadu, India
  • Designed and scaled cloud-native data pipelines using GCP (BigQuery, Dataflow, Pub/Sub, and Cloud Composer) to support near real-time analytics for global clients.
  • Built and optimized Snowflake and BigQuery data models for large retail and marketing datasets, improving query performance and cost efficiency.
  • Developed modular transformation logic using DBT, enabling faster onboarding of new data sources and easier long-term maintenance.
  • Implemented large-scale data processing using Python, Apache Beam, and Spark, ensuring stable performance as data volumes grew.
  • Built and deployed custom REST API connectors using Airbyte, running on Cloud Functions and Cloud Run for scalable and reliable data ingestion.
  • Tuned complex SQL queries and warehouse configurations to handle high-volume workloads while keeping compute costs under control.
  • Performed end-to-end data validation and reconciliation with source systems, consistently maintaining 99%+ data accuracy.
  • Used Gemini as a daily development assistant to accelerate Python, SQL, and DBT development, enabling faster feature delivery without compromising reliability.
  • Leveraged AI-generated suggestions to refactor transformations, optimize queries, and explore edge cases, improving code quality and long-term maintainability.
  • Applied AI support during pipeline debugging and validation, reducing investigation time and allowing greater focus on data accuracy and system design.
  • Collaborated remotely with product managers, analysts, and engineers to deliver data solutions aligned with business outcomes.

KPI Partners

Full-time · Hybrid

Data Engineer I

Jul 2022 – Jul 2023 · 1 yr · Hyderabad, Telangana, India
  • Built automated data pipelines using Apache Airflow, Python, and AWS, improving data availability and reducing manual reporting effort.
  • Designed scalable data models across BigQuery, Snowflake, and DynamoDB, supporting multi-user analytics and growing data volumes.
  • Developed reliable ETL workflows to process thousands of records per run with consistent performance.
  • Implemented data quality checks and monitoring to detect issues early and protect downstream analytics.
  • Worked closely with cross-functional teams to align data pipelines with reporting requirements and business needs.

DXC Technology

Full-time · Remote

Associate Professional Software Engineer

Aug 2021 - Jun 2022 · 11 mos · Bengaluru, Karnataka, India
  • Developed and maintained data pipelines using Azure Data Factory and Databricks, supporting enterprise-scale analytics workloads.
  • Optimized analytical storage using Azure Synapse Analytics and Delta Lake, improving query performance and reporting reliability.
  • Implemented secure data access using Azure Key Vault and role-based access control (RBAC), meeting enterprise security and compliance requirements.
  • Monitored and resolved pipeline issues using Azure Monitor and Log Analytics, reducing downtime and improving operational stability.
  • Tuned SQL queries and Spark jobs to ensure efficient execution on large datasets.

Cyber Crime Cell, Gurugram Police

Internship · Remote

Cyber Security Intern

Jun 2021 - Jul 2021 · 2 mos · Gurugram, Haryana, India
  • Provided first-level compliance monitoring and investigations.
  • Assisted with forensics analysis and fact gathering.
  • Assisted with vulnerability assessments and penetration testing for specific applications, services, networks, and servers as required.

Google

Internship · Remote

Google Cloud Ready Facilitator

Apr 2021 - Jun 2021 · 3 mos · Bengaluru, Karnataka, India
  • Learned Computing, Application Development, Big Data & Machine Learning using Google Cloud's training platform Qwiklabs.
  • Completed self-paced labs with temporary credentials to Google Cloud Platform for hands-on learning.

DXC Technology

Internship · Remote

Project Trainee

Feb 2021 - Apr 2021 · 3 mos · Bengaluru, Karnataka, India
  • The project aims to analyze and identify the data quality issues and strategies to mitigate these Issues in the given data. We need to start with a PowerPoint presentation that outlines the approach that we will be taking.
  • The client has agreed on a 3-week scope with the following 3 phases as follows - Data Exploration: Model Development
  • Created PowerPoint presentations outlining approaches for data quality improvement.

Featured Projects

Shopping Sales Project

Developed an end-to-end data pipeline using AWS services and Airflow for reporting from Google Sheets API data.

AWS Airflow Python ETL

Real-time Data Streaming

Implemented a real-time data ingestion system with Spark Streaming for processing high-velocity data.

Spark Kafka Scala Streaming

Flight Delay Prediction

Developed a machine learning model for predicting flight delays with 85% accuracy using historical data.

Python ML Pandas Scikit-learn

Customer Segmentation

Utilized R and clustering techniques to segment customers for targeted marketing campaigns.

R Clustering Data Analysis

Certifications

Google Cloud

Google Cloud Certified Professional Data Engineer

Microsoft Azure

Microsoft Certified: Fabric Analytics Engineer Associate

Microsoft Azure

Microsoft Certified: Azure Data Engineer Associate

AWS

AWS Certified Data Engineer - Associate

Microsoft Azure

Microsoft Certified: DevOps Engineer Expert

Microsoft Azure

Microsoft Certified: Azure Developer Associate

Microsoft Azure

Microsoft Certified: Azure Administrator Associate

AWS

AWS Certified Cloud Practitioner

My Resume

Get In Touch

durgaprasadchowdury@gmail.com

+91 9502192674

Pathapatnam, Andhra Pradesh, India

github.com/durgaprasadpr718
linkedin.com/in/durga-prasad-choudhury-75022314b