Skip to content

Commit b697c8a

Browse files
authored
Spackify JEDI components for NEPTUNE-JEDI (#1887)
This PR updates the JEDI packages from PR JCSDA/spack#395 that are relevant for NEPTUNE-JEDI (aka FALCON project) and adds them to our custom repo under repos/spack_stack. Building these components in spack-stack and providing them as modules allows us to reduce the NEPTUNE-JEDI build to a small number of repositories that are controlled and developed by NRL only, and - very important - allows us to implement a pure CMake build at the top level (neptune-bundle/neptune-ecosystem) without having to deal with ecbuild. A small change is made to the crtm package (in repos/builtin) to enable basic testing during spack install. Notes/Limitations The hashes added here are the exact versions needed by NEPTUNE-JEDI (aka FALCON), with additional bug fixes added on top of it in form of patches applied by Spack. In FALCON, these were kept in branches created from the commits used here. The Spack solution is cleaner, since it avoids maintaining branches. Most of these patches are already included in newer versions of the JEDI code. I made sure that all ctests that are enabled pass with GCC (tested gcc@13) and Intel oneAPI (tested oneapi@2025.3.0). I made sure that the tests that are disabled are not related to the Spack package, but to the code itself or other issues outside of the scope of this PR. To run these tests, use spack install --test=root ioda, for example. FALCON needs exactly these hashes in spack-stack-2.1.0 to transition from building JEDI components in neptune-bundle to loading them as modules. It is understood between JCSDA and NRL that these new Spack packages will not receive any additional support (i.e. beyond what JCSDA supports already) and that we will make an effort to update the JEDI code in the bundle to newer, official tags of the JEDI repositories. We cannot test the JEDI packages in CI yet, because the ropp-ufo code is coming from the NRL Enterprise GitHub because of the unresolved issue Stub with ROPP as submodule JCSDA-internal/ropp-ufo#1.
1 parent dd6740b commit b697c8a

20 files changed

Lines changed: 780 additions & 16 deletions

File tree

.github/workflows/ubuntu-ci-x86_64-gnu.yaml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -70,9 +70,10 @@ jobs:
7070
spack stack create env --site linux.default --template ${TEMPLATE} --name ${ENVNAME} --compiler gcc
7171
spack env activate ${ENVDIR}
7272
73-
# Turn off variant "adp" for neptune-dev (NRL internal only)
73+
# Turn off variants "adp" and "jedi" for neptune-dev (NRL internal only)
7474
if [[ "${TEMPLATE}" == *"neptune-dev"* ]]; then
7575
sed -i 's/+adp/~adp/g' ${ENVDIR}/spack.yaml
76+
sed -i 's/+jedi/~jedi/g' ${ENVDIR}/spack.yaml
7677
fi
7778
7879
unset SPACK_DISABLE_LOCAL_CONFIG

.github/workflows/ubuntu-ci-x86_64-oneapi-ifx.yaml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -70,9 +70,10 @@ jobs:
7070
spack stack create env --site linux.default --template ${TEMPLATE} --name ${ENVNAME} --compiler oneapi
7171
spack env activate ${ENVDIR}
7272
73-
# Turn off variant "adp" for neptune-dev (NRL internal only)
73+
# Turn off variants "adp" and "jedi" for neptune-dev (NRL internal only)
7474
if [[ "${TEMPLATE}" == *"neptune-dev"* ]]; then
7575
sed -i 's/+adp/~adp/g' ${ENVDIR}/spack.yaml
76+
sed -i 's/+jedi/~jedi/g' ${ENVDIR}/spack.yaml
7677
fi
7778
7879
unset SPACK_DISABLE_LOCAL_CONFIG

.github/workflows/ubuntu-ci-x86_64-oneapi.yaml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -70,9 +70,10 @@ jobs:
7070
spack stack create env --site linux.default --template ${TEMPLATE} --name ${ENVNAME} --compiler oneapi
7171
spack env activate ${ENVDIR}
7272
73-
# Turn off variant "adp" for neptune-dev (NRL internal only)
73+
# Turn off variants "adp" and "jedi" for neptune-dev (NRL internal only)
7474
if [[ "${TEMPLATE}" == *"neptune-dev"* ]]; then
7575
sed -i 's/+adp/~adp/g' ${ENVDIR}/spack.yaml
76+
sed -i 's/+jedi/~jedi/g' ${ENVDIR}/spack.yaml
7677
fi
7778
7879
unset SPACK_DISABLE_LOCAL_CONFIG

configs/common/packages_gcc.yaml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,3 +23,6 @@ packages:
2323
ectrans:
2424
require:
2525
- +fftw
26+
oops:
27+
require:
28+
- ~mkl

configs/common/packages_oneapi.yaml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,9 @@ packages:
5151
nco:
5252
require:
5353
- '%c,cxx=gcc'
54+
oops:
55+
require:
56+
- +mkl
5457
parallel-netcdf:
5558
require:
5659
- +shared-intel

configs/templates/neptune-dev-llvm/spack.yaml

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -19,10 +19,7 @@ spack:
1919
#- neptune-python-env ^neptune-env ~debug +openmp +espc +ncview ^esmf@=8.9.1
2020
#- jedi-neptune-env +adp ^neptune-env ~debug +openmp +espc +ncview ^esmf@=8.9.1
2121
- py-numpy
22-
# Let's assume/hope that by the time FALCON starts
23-
# testing with LLVM, we have moved from crtm v2 to v3
24-
- crtm@3.1.2
25-
#- crtm@v2.4.1-jedi.2
22+
- crtm@3.1.3
2623

2724
packages:
2825
# Turn on python variant for esmf

configs/templates/neptune-dev/spack.yaml

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -13,9 +13,8 @@ spack:
1313
- neptune-env ~debug ~openmp +espc +ncview ^esmf@=8.9.1
1414
- neptune-env +debug ~openmp +espc +ncview ^esmf@=8.9.1
1515
- neptune-python-env +gittools ^neptune-env ~debug +openmp +espc +ncview ^esmf@=8.9.1
16-
- jedi-neptune-env +adp ^neptune-env ~debug +openmp +espc +ncview ^esmf@=8.9.1
17-
- crtm@3.1.2
18-
- crtm@v2.4.1-jedi.2
16+
- jedi-neptune-env +adp +jedi ^neptune-env ~debug +openmp +espc +ncview ^esmf@=8.9.1
17+
- crtm@3.1.3
1918

2019
packages:
2120
# Turn on python variant for esmf

configs/templates/unified-dev/spack.yaml

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -11,9 +11,10 @@ spack:
1111
- ewok-env
1212
#- ai-env
1313
- geos-gcm-env ^esmf@=8.9.1
14-
- global-workflow-env ^esmf@=8.6.1 ^crtm@=3.1.2
14+
- global-workflow-env ^esmf@=8.6.1 ^crtm@=3.1.3
15+
# Is GMAO ready to move to crtm@3.1.3?
1516
- gmao-swell-env ^crtm@=v2.4.1-jedi.2
16-
- gsi-env ^crtm@=3.1.2
17+
- gsi-env ^crtm@=3.1.3
1718
- jedi-fv3-env
1819
- jedi-geos-env ^esmf@=8.9.1
1920
- jedi-mpas-env
@@ -24,12 +25,12 @@ spack:
2425
- neptune-env ^esmf@=8.9.1
2526
- neptune-python-env ^esmf@=8.9.1
2627
- soca-env
27-
- ufs-srw-app-env ^esmf@=8.8.0 ^crtm@=3.1.2
28-
- ufs-weather-model-env ^esmf@=8.8.0 ^crtm@=3.1.2
28+
- ufs-srw-app-env ^esmf@=8.8.0 ^crtm@=3.1.3
29+
- ufs-weather-model-env ^esmf@=8.8.0 ^crtm@=3.1.3
2930

3031
# Various crtm tags (list all to avoid duplicate packages)
3132
- crtm@v2.4.1-jedi.2
32-
- crtm@3.1.2
33+
- crtm@3.1.3
3334

3435
# Mapl with various esmf tags
3536
- mapl@2.53.0 ^esmf@8.6.1
Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
--- a/cmake/ioda-import.cmake.in
2+
+++ b/cmake/ioda-import.cmake.in
3+
@@ -65,6 +65,15 @@
4+
if(@odc_FOUND@)
5+
find_dependency( odc 1.0.2 REQUIRED )
6+
endif()
7+
+
8+
+if(@bufr_FOUND@)
9+
+ find_dependency( bufr 12.0.1 REQUIRED )
10+
+endif()
11+
+
12+
+if(@bufr_query_FOUND@)
13+
+ find_dependency( bufr_query 0.0.4 REQUIRED )
14+
+endif()
15+
+
16+
# Header-only. Not exposed.
17+
#find_dependency( Boost 1.64.0 )
18+
Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
# Copyright 2013-2022 Lawrence Livermore National Security, LLC and other
2+
# Spack Project Developers. See the top-level COPYRIGHT file for details.
3+
#
4+
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
5+
6+
from spack_repo.builtin.build_systems.cmake import CMakePackage, generator
7+
8+
from spack.package import *
9+
10+
11+
class Ioda(CMakePackage):
12+
"""Interface for Observation Data Access"""
13+
14+
homepage = "https://github.com/JCSDA/ioda"
15+
git = "https://github.com/JCSDA/ioda.git"
16+
17+
maintainers("climbfuji")
18+
19+
version("develop", branch="develop", no_cache=True)
20+
version("2.9.0.20250826", commit="6e76616001067384f7d0ca4341ad78e81527af8b")
21+
22+
patch("ioda_cmake_import.patch", when="@2.9.0.20250826")
23+
24+
variant("doc", default=False, description="Build IODA documentation")
25+
# Let's always assume IODA_BUILD_LANGUAGE_FORTRAN=on.
26+
# variant('fortran', default=True, description='Build the ioda Fortran interface')
27+
variant("odc", default=True, description="Build ODC bindings")
28+
# ioda has no explicit OpenMP calls, but header files from Eigen and oops do use openmp.
29+
variant("openmp", default=True, description="Build with OpenMP support")
30+
# Let's always BUILD_PYTHON_BINDINGS.
31+
# variant('python', default=True, description='Build the ioda Python interface')
32+
33+
generator("make")
34+
35+
# Project doesn't list "c" as a dependency in CMakeLists.txt, but cmake step fails w/o it
36+
depends_on("c", type=("build"))
37+
depends_on("cxx", type=("build"))
38+
depends_on("fortran", type=("build"))
39+
40+
depends_on("boost@1.64.0:")
41+
depends_on("bufr")
42+
depends_on("bufr@12.0.1:", when="@2.9:")
43+
depends_on("bufr-query@0.0.4:", when="@2.9:")
44+
depends_on("cmake", type=("build"))
45+
depends_on("cmake@3.14:", type=("build"), when="@2.9:")
46+
depends_on("ecbuild", type=("build"))
47+
depends_on("ecbuild@3.3.2:", type=("build"), when="@2.9:")
48+
depends_on("eckit")
49+
depends_on("eckit@1.23.0:", when="@2.9:")
50+
depends_on("eigen")
51+
depends_on("fckit")
52+
depends_on("fckit@0.10.1:", when="@2.9:")
53+
depends_on("gsl-lite")
54+
depends_on("hdf5@1.12.0: +mpi")
55+
depends_on("hdf5@1.14.0: +mpi", when="@2.9:")
56+
depends_on("ioda-data", type=("build", "test"))
57+
depends_on("ioda-data@2.9.0.20250805", type=("build", "test"), when="@2.9:")
58+
depends_on("jedi-cmake", type=("build"))
59+
depends_on("llvm-openmp", when="+openmp %apple-clang", type=("build", "link", "run"))
60+
depends_on("mpi")
61+
depends_on("nccmp", type=("build", "test"))
62+
depends_on("netcdf-cxx", when="@2.9:")
63+
depends_on("odc", when="+odc")
64+
depends_on("odc@1.4.6:", when="@2.9: +odc")
65+
depends_on("oops+openmp", when="+openmp")
66+
depends_on("oops~openmp", when="~openmp")
67+
depends_on("oops@1.10", when="@2.9:")
68+
depends_on("python")
69+
depends_on("python@3.9:3.11", when="@2.9:")
70+
depends_on("py-pybind11")
71+
depends_on("py-pycodestyle", type=("build", "test"))
72+
depends_on("py-netcdf4", type=("build", "test"))
73+
depends_on("udunits")
74+
depends_on("udunits@2.2.0:", when="@2.9:")
75+
76+
def cmake_args(self):
77+
res = [
78+
self.define("BUILD_TESTING", self.run_tests),
79+
self.define_from_variant("ENABLE_IODA_DOC", "doc"),
80+
]
81+
return res
82+
83+
def setup_build_environment(self, env: EnvironmentModifications) -> None:
84+
"""This needs to be set at build time, not at test time,
85+
to prevent IODA from downloading test data from S4"""
86+
env.set("IODA_TESTFILES", self.spec["ioda-data"].prefix)
87+
88+
def check(self):
89+
skipped_tests = None
90+
with when("@2.9.0.20250826"):
91+
# No time to deal with the bufr Python dependency
92+
skipped_tests = [
93+
"test_ioda_bufr_python_encoder",
94+
"test_ioda_bufr_python_parallel",
95+
]
96+
97+
ctest = Executable(self.spec["cmake"].prefix.bin.ctest)
98+
with working_dir(self.build_directory):
99+
if skipped_tests:
100+
ctest("--timeout", "120", "-E", "|".join(skipped_tests))
101+
else:
102+
ctest("--timeout", "120")
103+
104+
@run_after("install")
105+
def fix_ioda_yaml_root_path(self):
106+
with when("@2.9.0.20250826"):
107+
filter_file(
108+
join_path(self.build_directory, "share/test/testinput/"),
109+
join_path(self.prefix, "share/ioda/yaml"),
110+
join_path(self.prefix, "lib64/cmake/ioda/ioda-import.cmake"),
111+
)

0 commit comments

Comments
 (0)