24 AI Prompts for Data Engineers

You are a senior data engineer specializing in {pipeline_tool}. Create a complete data pipeline to extract {data_type} from {source_system} in {data_format} format, transform it according to {transformation_requirements}, and load it into {target_platform} with {frequency} processing. Include error handling, data validation, monitoring alerts, and provide the full code with configuration files and deployment instructions.

You are a data pipeline optimization expert. Analyze and redesign an existing {pipeline_tool} pipeline that processes {data_type} from {source_system} to {target_platform} but currently fails to meet {performance_constraint}. Identify bottlenecks, recommend specific optimizations including partitioning strategies, parallel processing, and resource allocation, then provide the optimized code with before/after performance comparisons.

You are a data integration architect. Design a unified pipeline using {pipeline_tool} that ingests {data_type} from multiple sources: {source_system}, plus two additional disparate systems of your choice. Handle schema differences, implement data quality checks, create a master data model for {target_platform}, and ensure {frequency} synchronization with conflict resolution logic and comprehensive logging.

You are a data quality analyst examining a {data_source} containing {data_type} with approximately {data_volume}. Analyze the provided sample data to identify the top 5 most critical quality issues, categorize them by severity (Critical/High/Medium), and estimate the percentage of records affected by each issue. Provide a prioritized action plan with specific steps to address each problem, including recommended tools or techniques for resolution.

You are designing automated validation rules for {data_type} in the {industry} industry. Based on the provided data schema and business requirements, create a comprehensive validation framework that includes: (1) field-level validation rules with specific criteria, (2) cross-field dependency checks, (3) business logic constraints, and (4) data quality scoring methodology. Format your response as implementable validation rules with clear pass/fail conditions and error messages.

You are a data engineer tasked with cleaning and standardizing {data_type} from {data_source} that contains {quality_issues}. Develop a step-by-step data cleaning pipeline that includes: automated detection methods, standardization rules, duplicate resolution strategies, and missing value handling approaches. Provide specific transformation logic, validation checkpoints, and quality metrics to measure improvement, ensuring the cleaned data meets {business_rules} requirements.

You are a database architect analyzing raw data to create an optimal schema. Given this {data_source} containing {data_sample}, design a normalized schema for {target_system} that serves a {business_domain} application. Provide the complete DDL statements, identify primary/foreign keys, suggest appropriate indexes for {performance_requirements}, and explain your normalization decisions and any assumptions about data relationships.

You are a database migration specialist managing schema changes for a production {target_system} database. I need to modify {existing_schema} to accommodate {migration_type} while handling {data_volume}. Create a step-by-step migration plan including: backward-compatible DDL changes, data transformation scripts, rollback procedures, and validation queries to ensure data integrity throughout the process.

You are a data migration engineer converting schemas between database systems. Transform this {existing_schema} from {source_system} to {target_system}, accounting for platform-specific differences in data types, constraints, and features. Provide the converted schema, highlight any functionality gaps or required workarounds, and suggest optimization opportunities specific to the target platform for {business_domain} workloads.

You are a data catalog specialist analyzing {data_source_type} from the {business_domain} domain at {organization_name}. Create comprehensive catalog entries by examining the provided schema and sample data. Generate technical metadata (columns, types, relationships), business-friendly descriptions, data quality indicators, and discoverable tags. Structure your response with: Asset Overview, Schema Documentation, Business Context, Quality Assessment, and Access Patterns for maximum usability by downstream {user_role} users.

You are a data steward enhancing existing catalog entries for {data_format} containing {business_domain} data. Analyze the current entry and enrich it with clear business definitions, practical usage examples, data lineage details, and quality metrics. Add user-friendly search tags and explanations that help {user_role} users quickly understand both technical specifications and business value. Focus on filling documentation gaps that prevent effective data discovery.

You are a data discovery assistant helping a {user_role} in {industry} find datasets for {business_domain} analysis. Search available catalog entries and recommend the most relevant datasets, explaining why each matches their needs. Include data freshness, access methods, known limitations, and suggest related datasets they might find valuable. Prioritize recommendations that enable immediate productive use while highlighting any preparation requirements.

You are a database performance expert analyzing a slow query. Given this {database_system} query that currently takes {execution_time} to execute: `{query}`. The query involves {table_names} with approximately {data_volume} records each. Analyze the query structure and provide 3-5 specific optimization recommendations ranked by impact, including rewritten query examples and expected performance improvements.

Act as a database architect reviewing indexing strategy for {database_system}. Here's a query: `{query}` running against tables {table_names} with {data_volume} records. Analyze the current execution plan and recommend optimal index configurations, including composite indexes, covering indexes, and any schema adjustments. Provide the exact CREATE INDEX statements and explain the performance rationale for each recommendation.

You are a senior database developer tasked with refactoring this complex {database_system} query for better performance: `{query}`. The business requirement is {business_context} and current execution time is {execution_time}. Suggest alternative query approaches (CTEs, temp tables, query decomposition) and provide 2-3 rewritten versions with explanations of when to use each approach based on data patterns and system constraints.

You are a data governance specialist conducting a systematic data discovery assessment. Analyze our {systems} environment containing {data_types} and create a prioritized classification framework based on {regulations} requirements. Provide a step-by-step discovery plan that identifies high-risk data locations, recommends appropriate classification labels (public, internal, confidential, restricted), and suggests immediate containment actions for any exposed sensitive data in our {industry} organization.

You are a cybersecurity analyst reviewing access permissions for {data_types} across {systems} in a {organization_size} {industry} company. Evaluate our current access controls against the principle of least privilege and {regulations} requirements. Generate a detailed remediation plan that identifies over-privileged users, recommends role-based access improvements, and provides specific steps to implement proper segregation of duties while maintaining business functionality.

You are a compliance officer preparing for a {regulations} audit with a {timeline} deadline. Assess our current data governance practices for {data_types} stored in {systems} and identify critical compliance gaps. Create an actionable remediation roadmap that prioritizes high-risk violations, provides specific policy updates needed, includes required documentation templates, and establishes monitoring procedures to maintain ongoing compliance in our {industry} environment.

You are a real-time data engineering expert designing a streaming analytics system. Create a comprehensive architecture plan for processing {data_source} (e.g., IoT temperature sensors, financial transactions, web clickstreams) using {streaming_platform} (e.g., Apache Kafka, AWS Kinesis, Google Pub/Sub). The system needs to handle {data_volume} (e.g., 10,000 events/second, 1GB/hour) with {latency_requirement} (e.g., sub-second, under 5 minutes) processing. Include specific technology recommendations, data flow diagrams, and implementation steps for {business_use_case} (e.g., predictive maintenance, fraud detection, real-time personalization).

You are a data scientist implementing real-time anomaly detection for {business_domain} (e.g., e-commerce website performance, manufacturing equipment, financial trading). Design an AI-driven alerting system that monitors {key_metrics} (e.g., response time, temperature readings, transaction amounts) and detects {anomaly_types} (e.g., sudden spikes, gradual drift, pattern breaks). Create specific algorithms, threshold definitions, and alert rules that minimize false positives while catching critical issues within {time_window} (e.g., 30 seconds, 2 minutes). Include escalation procedures and integrate with {notification_system} (e.g., Slack, PagerDuty, email).

You are a systems architect optimizing a real-time data processing pipeline that currently handles {current_load} (e.g., 5,000 events/second, 500MB/hour) but needs to scale to {target_load} (e.g., 50,000 events/second, 5GB/hour). The system processes {data_type} (e.g., JSON logs, binary sensor data, CSV files) using {current_stack} (e.g., Apache Storm + Redis, Spark Streaming + Cassandra). Identify specific bottlenecks, recommend architectural improvements, and create a step-by-step scaling plan that maintains {performance_requirements} (e.g., 99.9% uptime, <100ms latency) while optimizing costs for {cloud_platform} (e.g., AWS, Google Cloud, Azure).

You are a cloud cost optimization specialist conducting a monthly review. Analyze my {cloud_provider} data infrastructure spending of ${monthly_budget} across {primary_services} handling {data_volume} of data with {usage_pattern} workloads. Identify the top 5 cost optimization opportunities, provide specific implementation steps for each, and estimate potential monthly savings with associated risks or trade-offs.

You are a cloud architect tasked with immediate cost reduction. My {cloud_provider} bill unexpectedly increased to ${monthly_budget} this month, primarily from {primary_services} processing {data_volume} of data. I need to reduce costs by 30-40% within {time_frame} while maintaining {performance_requirements}. Provide 3 immediate actions I can take today, 2 medium-term optimizations for this week, and quantify the expected savings for each.

You are designing a cost-optimized data architecture. Help me restructure my current {cloud_provider} setup handling {data_volume} with {usage_pattern} workloads under ${monthly_budget} budget, considering {business_constraints} requirements. Compare 3 architectural approaches (current vs. 2 alternatives), detail the migration effort for each, and project 12-month total cost of ownership including implementation costs.