Feat/mp package

[echo-chamber0]

Add GCP Marketplace deployment package (Terraform + Helm)

Summary

• Add mp-pkg/ — complete GCP Marketplace deployment package for Data Commons Accelerator, containing Terraform module and Helm chart for deployment via GCP Marketplace
• Restructure docs/ — condense and improve deployment guide and user guide with better navigation, standardized formatting, and operational tips

What's included

mp-pkg/terraform/ — Infrastructure as Code

• Terraform configuration for GCP Marketplace
• CloudSQL MySQL with private IP (via Private Service Access)
• GCS bucket for custom data storage
• Workload Identity (GKE SA ↔ GCP SA binding)
• Google Maps API keys provisioning
• Kubernetes secrets management
• Helm-based application deployment
• 4 reusable modules: cloudsql, gcs-bucket, k8s-secrets, maps-api-keys
• Marketplace metadata: metadata.yaml, metadata.display.yaml, schema.yaml

mp-pkg/charts/datacommons/ — Helm Chart (v3.3.12)

• Application Deployment configuration
• Database initialization Job (pre-install hook)
• Database sync CronJob (periodic data import)
• Application CRD for Marketplace integration
• Workload Identity ServiceAccount binding

docs/ — Documentation updates

• Restructured Deployment Guide with TOC, GitHub alerts, and standardized [BRACKETED] placeholders
• Improved User Guide with data sync CronJob tips and centralized common workflows

[gemini-code-assist]

Summary of Changes

Hello @echo-chamber0, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the deployability of the Data Commons Accelerator by introducing a comprehensive GCP Marketplace package. This package streamlines the setup process by automating infrastructure provisioning and application deployment, making it easier for users to get a Data Commons instance running on GKE with integrated GCP services. The accompanying documentation updates further improve the user experience by providing clearer, more structured guidance.

Highlights

• GCP Marketplace Deployment Package: Introduced a complete GCP Marketplace deployment package under mp-pkg/, which includes both Terraform modules for infrastructure provisioning and a Helm chart for application deployment.
• Terraform Infrastructure as Code: Developed a Terraform module to provision essential GCP resources such as CloudSQL MySQL with private IP, GCS buckets for data storage, Workload Identity for secure access, and Google Maps API keys, along with Kubernetes secrets management.
• Helm Chart for Application Deployment: Created a Helm chart (v3.3.12) for the Data Commons Accelerator application, encompassing deployment configurations, a pre-install hook for database initialization, a CronJob for periodic data synchronization, and an Application CRD for Marketplace integration.
• Documentation Restructuring and Improvement: Restructured and enhanced the docs/ directory, specifically the Deployment Guide and User Guide, to improve navigation, standardize formatting, and add operational tips and GitHub alerts for better clarity.

Changelog

• docs/DEPLOYMENT_GUIDE.md
  • Updated the description of provisioned infrastructure to include Cloud Storage Bucket and Kubernetes Workload twice, likely a copy-paste error.
  • Modified GKE cluster creation guidance to use a GitHub-style NOTE alert.
  • Added a redundant heading for 'Step 2: Complete the Deployment Configuration Form'.
  • Converted a plain text instruction about tooltips into a GitHub-style TIP alert.
  • Replaced a section on finding existing PSA ranges with a detailed GitHub-style CAUTION alert, including commands to check for existing PSA and GKE VPC network names.
  • Converted a warning about closing the browser tab during deployment into a GitHub-style WARNING alert.
  • Transformed the deployment outputs access information into a GitHub-style TIP alert with expanded details and direct links.
  • Standardized NAMESPACE placeholders in kubectl commands to [NAMESPACE] for consistency.
  • Updated gcloud container clusters get-credentials command placeholders to [CLUSTER_NAME], [LOCATION], and [PROJECT_ID].
  • Changed the local access URL to use angle brackets for clarity.
  • Added a blank line before 'Common deployment errors' and 'Common pod issues' for better readability.
  • Standardized NAMESPACE and POD_NAME placeholders in kubectl diagnostic commands to [NAMESPACE] and [POD_NAME].
  • Added a duplicate comment line '# Check existing PSA ranges'.
  • Converted the explanation for port-forward connection drops into a GitHub-style NOTE alert.
  • Standardized NAMESPACE placeholder in kubectl get pods command to [NAMESPACE].
  • Converted the deletion instruction into a GitHub-style IMPORTANT alert, emphasizing cost savings.
• docs/USER_GUIDE.md
  • Added a Table of Contents for improved navigation.
  • Updated the 'Getting Started' section with a TIP alert linking to the Deployment Guide.
  • Added a TIP alert for retrieving administrator credentials, directing users to Infrastructure Manager Deployments.
  • Modified the kubectl get secret command for retrieving admin credentials to a more direct jsonpath query.
  • Clarified placeholders in kubectl commands for admin credential retrieval.
  • Updated the 'Administrator Log In' steps for clarity and consistency.
  • Added a new 'Upload Custom Data' section with detailed steps and GitHub-style NOTE and TIP alerts for data processing and manual sync.
  • Updated the 'Customize User Interface' section to use 'Organization Branding' instead of 'University Branding'.
  • Revised the 'Data Security' section to specify data residence in a GCS bucket within the GCP environment and added a NOTE alert.
  • Refactored domain-specific sections (Education, Health, Energy) to use consistent subheadings like 'Overview', 'For Administrators', and 'For Data Analysts & Researchers'.
  • Updated formatting for bullet points in various sections to use hyphens instead of asterisks.
  • Added an IMPORTANT alert in the 'Energy: For Administrators' section regarding CSV data requirements.
  • Added a TIP alert in the 'Known Limitations' section, directing users to the Deployment Guide for troubleshooting.
• mp-pkg/charts/datacommons/.helmignore
  • Added a new file to specify patterns for files and directories to be ignored when packaging the Helm chart, including common version control, backup, IDE, testing, documentation, and CI/CD files.
• mp-pkg/charts/datacommons/Chart.yaml
  • Added a new Helm chart definition file, specifying API version, name, description, type, version, appVersion, GCP Marketplace annotations, and chart metadata like icon and keywords.
• mp-pkg/charts/datacommons/crds/app-crd.yaml
  • Added a new Custom Resource Definition (CRD) for Kubernetes Applications (app.k8s.io/v1beta1), defining its schema, additional printer columns, and status fields.
• mp-pkg/charts/datacommons/templates/NOTES.txt
  • Added a new Helm template for post-installation notes, providing instructions for quick status checks, accessing the application via port-forward or Cloud Shell, configuring Ingress, monitoring, and troubleshooting.
• mp-pkg/charts/datacommons/templates/_helpers.tpl
  • Added a new Helm template file containing helper definitions for generating chart names, full names, common labels, selector labels, service account names, GCS input/output directories, and namespace.
• mp-pkg/charts/datacommons/templates/application.yaml
  • Added a new Helm template for the Kubernetes Application CRD, defining its metadata, descriptor (type, version, description, links, notes), selector, and component kinds.
• mp-pkg/charts/datacommons/templates/configmap.yaml
  • Added a new Helm template for a Kubernetes ConfigMap, configuring various application settings such as Data Commons API root, CloudSQL, GCS, Natural Language features, GCP project ID, Flask environment, Mixer settings, and debug options.
• mp-pkg/charts/datacommons/templates/db-init-job.yaml
  • Added a new Helm template for a Kubernetes Job (db-init) to perform database initialization, including image configuration, environment variables from ConfigMap and Secrets, and resource limits.
• mp-pkg/charts/datacommons/templates/db-sync-cronjob.yaml
  • Added a new Helm template for a Kubernetes CronJob (db-sync) to periodically synchronize database data, defining its schedule, job history limits, concurrency policy, image, environment variables, and resource limits.
• mp-pkg/charts/datacommons/templates/deployment.yaml
  • Added a new Helm template for the main application Deployment, configuring replicas, container image, ports, environment variables, resource limits, health probes (startup, readiness, liveness), and security contexts.
• mp-pkg/charts/datacommons/templates/secret.yaml
  • Added a new Helm template for a Kubernetes Secret, managing admin panel credentials by either preserving existing ones on upgrade or generating new ones on first install.
• mp-pkg/charts/datacommons/templates/service.yaml
  • Added a new Helm template for the Kubernetes Service, defining its type, ports, and selector to expose the application.
• mp-pkg/charts/datacommons/templates/serviceaccount.yaml
  • Added a new Helm template for the Kubernetes ServiceAccount, including annotations for Workload Identity binding to a GCP Service Account.
• mp-pkg/charts/datacommons/values.yaml
  • Added a new Helm values file, providing default configurations for deployment, database initialization, database synchronization, application settings (CloudSQL, GCS, Natural Language), service account, and service exposure.
• mp-pkg/terraform/README.md
  • Added a new README file detailing the Terraform infrastructure for Data Commons Accelerator, covering enterprise flexibility features, requirements, deployed resources, Private Service Access configuration, and input/output variables.
• mp-pkg/terraform/api-keys.tf
  • Added a new Terraform configuration file to manage Google Maps API keys, including API targets and labels.
• mp-pkg/terraform/cloudsql.tf
  • Added a new Terraform configuration file to provision a CloudSQL MySQL instance with private IP, manage Private Service Access (PSA) global addresses and service networking connections, and validate existing PSA configurations.
• mp-pkg/terraform/gcs.tf
  • Added a new Terraform configuration file to create a GCS bucket for Data Commons data, grant storage.objectAdmin IAM role to the workload service account, and create default input/ and output/ directories.
• mp-pkg/terraform/helm.tf
  • Added a new Terraform configuration file to manage the Helm release of the Data Commons application, setting various chart values dynamically based on Terraform variables for container images, application configuration, resource allocation, CloudSQL, GCS, Workload Identity, and secrets.
• mp-pkg/terraform/locals.tf
  • Added a new Terraform file defining local variables for common labels, resource presets, computed values (region, resource names with overrides), namespace derivation, VPC network details, and PSA configuration.
• mp-pkg/terraform/main.tf
  • Added a new main Terraform configuration file, setting up Google Cloud, Kubernetes, and Helm providers, defining required GCP API services, and retrieving GKE cluster and VPC network details.
• mp-pkg/terraform/marketplace_test.tfvars
  • Added a new Terraform variable file (marketplace_test.tfvars) for validating the Terraform template in GCP Marketplace, providing example values for deployment, GKE, CloudSQL, GCS, API keys, and application configuration.
• mp-pkg/terraform/metadata.display.yaml
  • Added a new YAML file defining the UI display metadata for the Data Commons Accelerator in GCP Marketplace, including variable definitions with titles, tooltips, validations, placeholders, and section organization.
• mp-pkg/terraform/metadata.yaml
  • Added a new YAML file defining the blueprint metadata for the Data Commons Accelerator in GCP Marketplace, including title, version, actuation tool, detailed description, architecture, and interfaces for input/output variables.
• mp-pkg/terraform/modules/cloudsql/main.tf
  • Added a new Terraform module for creating a Google CloudSQL MySQL instance, configuring its settings (tier, availability, storage, backup, network, maintenance, flags, insights), and creating a database and user with a randomly generated password.
• mp-pkg/terraform/modules/cloudsql/outputs.tf
  • Added a new Terraform module defining outputs for the CloudSQL instance, including instance name, connection details, network information, database name, user credentials (sensitive), connection strings, and Kubernetes environment variables/secret data.
• mp-pkg/terraform/modules/cloudsql/variables.tf
  • Added a new Terraform module defining input variables for the CloudSQL instance, covering project ID, region, instance name, network links, database version, tier, zones, availability type, disk configuration, backup settings, database/user names, deletion protection, and labels.
• mp-pkg/terraform/modules/cloudsql/versions.tf
  • Added a new Terraform module specifying required Terraform and provider versions for the CloudSQL module.
• mp-pkg/terraform/modules/gcs-bucket/main.tf
  • Added a new Terraform module for creating a GCS bucket, configuring its name, project, location, storage class, force destroy, labels, uniform bucket-level access, optional versioning, encryption, and lifecycle rules, along with IAM member bindings.
• mp-pkg/terraform/modules/gcs-bucket/outputs.tf
  • Added a new Terraform module defining outputs for the GCS bucket, including bucket name, URL, self-link, and the full bucket resource object.
• mp-pkg/terraform/modules/gcs-bucket/variables.tf
  • Added a new Terraform module defining input variables for the GCS bucket, covering project ID, bucket name, location, storage class, versioning, lifecycle rules, IAM members, encryption key, force destroy, and labels.
• mp-pkg/terraform/modules/gcs-bucket/versions.tf
  • Added a new Terraform module specifying required Terraform and provider versions for the GCS bucket module.
• mp-pkg/terraform/modules/k8s-secrets/main.tf
  • Added a new Terraform module for creating Kubernetes Secrets, iterating through a map of secret definitions to create kubernetes_secret_v1 resources with specified names, namespaces, labels, data, and type.
• mp-pkg/terraform/modules/k8s-secrets/outputs.tf
  • Added a new Terraform module defining outputs for the Kubernetes Secrets, including maps of secret names, namespaces, and full resource IDs.
• mp-pkg/terraform/modules/k8s-secrets/variables.tf
  • Added a new Terraform module defining input variables for Kubernetes Secrets, covering namespace, a map of secret definitions (data and optional labels), secret type, and common labels.
• mp-pkg/terraform/modules/k8s-secrets/versions.tf
  • Added a new Terraform module specifying required Terraform and provider versions for the Kubernetes Secrets module.
• mp-pkg/terraform/modules/maps-api-keys/main.tf
  • Added a new Terraform module for creating Google Maps API keys, including a random suffix for unique naming, API targets, and optional browser, Android, and iOS application restrictions.
• mp-pkg/terraform/modules/maps-api-keys/outputs.tf
  • Added a new Terraform module defining outputs for the Google Maps API key, including its ID, name, UID, the actual key value (sensitive), and Kubernetes secret data.
• mp-pkg/terraform/modules/maps-api-keys/variables.tf
  • Added a new Terraform module defining input variables for Google Maps API keys, covering project ID, name prefix, API targets, and allowed referrers/applications for restrictions.
• mp-pkg/terraform/modules/maps-api-keys/versions.tf
  • Added a new Terraform module specifying required Terraform and provider versions for the Google Maps API keys module.
• mp-pkg/terraform/outputs.tf
  • Added a new Terraform file defining overall deployment outputs, including Kubernetes namespace, GCS bucket URL, and various kubectl and gcloud commands for configuration, verification, port-forwarding, Cloud Shell access, data upload, log viewing, and admin credential retrieval.
• mp-pkg/terraform/schema.yaml
  • Added a new YAML file defining the GCP Marketplace image mapping schema, specifying variables for CDC Services and Data service image repositories and tags.
• mp-pkg/terraform/secrets.tf
  • Added a new Terraform configuration file to manage Kubernetes namespaces (conditional creation/data source for existing) and integrate with the k8s-secrets module to create Kubernetes secrets for database passwords, Maps API keys, and Data Commons API keys.
• mp-pkg/terraform/variables.tf
  • Added a new Terraform file defining all input variables for the deployment, including Marketplace-specific variables, project/region configuration, GKE cluster details, PSA network settings, CloudSQL configuration, GCS bucket settings, resource name overrides, API keys, application configuration, and container image/Helm chart variables.
• mp-pkg/terraform/versions.tf
  • Added a new Terraform file specifying the required Terraform version and provider versions for Google Cloud, Kubernetes, Helm, and Random providers.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a comprehensive GCP Marketplace deployment package for the Data Commons Accelerator, including Terraform modules and a Helm chart. It also restructures the documentation for better clarity. While the infrastructure code is well-structured and follows best practices for Marketplace solutions (e.g., using Workload Identity and Private Service Access), there are several duplicate lines in the documentation and a potential connectivity issue in the CloudSQL configuration regarding SSL mode. The review focuses on these correctness and maintainability issues.

[echo-chamber0]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a comprehensive GCP Marketplace deployment package for the Data Commons Accelerator, automating the provisioning of infrastructure like a GKE cluster, VPC, CloudSQL database, and GCS bucket using Terraform and Helm. While the code quality is excellent and follows modern security best practices, a critical security concern was identified: the lack of application restrictions on the generated Google Maps API key, which needs to be addressed to prevent unauthorized usage. Additionally, there is a minor suggestion to improve clarity in the Helm chart's configuration.