Introduce formal principle documentation to support contrib changes #3768

mrmundt · 2025-10-21T23:03:45Z

Fixes None but starts to address #2460

Summary/Motivation:

Stage 1 of the changes inside pyomo.contrib is documenting a lot of different development principles that make the change a lot more reasonably understandable. This PR creates the new directories with their basic READMEs as well as creates documentation about our overarching development principles, including a teaser to the upcoming changes in contrib as part of their contents.

Changes proposed in this PR:

Create basic structure for addons and devel
Create new principles.rst documentation
Update the testing guidance because it was missing something kind of essential
(Tiny extra fix) Include doc/OnlineDocs/api in .gitignore because I have almost accidentally committed those files WAY too many times

Legal Acknowledgement

By contributing to this software project, I have read the contribution guide and agree to the following terms and conditions for my contribution:

I agree my contributions are submitted under the BSD license.
I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

codecov · 2025-10-22T01:25:29Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.45%. Comparing base (b404bca) to head (92b607c).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3768   +/-   ##
=======================================
  Coverage   89.44%   89.45%           
=======================================
  Files         905      905           
  Lines      105485   105485           
=======================================
+ Hits        94355    94362    +7     
+ Misses      11130    11123    -7

Flag	Coverage Δ
builders	`29.14% <100.00%> (-0.01%)`	⬇️
default	`86.07% <100.00%> (?)`
expensive	`35.76% <100.00%> (?)`
linux	`86.76% <100.00%> (-2.46%)`	⬇️
linux_other	`86.76% <100.00%> (+<0.01%)`	⬆️
osx	`82.91% <100.00%> (ø)`
win	`84.99% <100.00%> (+<0.01%)`	⬆️
win_other	`84.99% <100.00%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

doc/OnlineDocs/principles.rst

blnicho

I have a bunch of minor edit suggestions but overall I think this looks great!

doc/OnlineDocs/contribution_guide.rst

doc/OnlineDocs/principles.rst

pyomo/addons/README.md

pyomo/devel/example/__init__.py

michaelbynum

I'm only about halfway done, but here are a couple comments/questions.

doc/OnlineDocs/contribution_guide.rst

michaelbynum

I have some minor comments, but I think this looks great overall.

michaelbynum · 2025-11-03T23:42:05Z

doc/OnlineDocs/principles.rst

+change until the next major Pyomo release, which will coincide with a
+new edition of the book.


While this is what we currently do, it is a strong statement/commitment. I'm fine with this, but I wanted to flag it for discussion.

Yeah, I might be for hedging and just saying next major Pyomo release and failing to mention the promise of books?

We talked about this in the dev call today (loosely). This is our "current" approach but nothing says we can't change it in the future, especially after the next book.

We can always hedge our promise with something like will hopefully coincide

michaelbynum · 2025-11-03T23:44:03Z

doc/OnlineDocs/principles.rst

+
+Functionality that is part of the Pyomo source tree but not explicitly
+included in the book is also expected to be stable if it resides outside
+the ``contrib`` (or future ``addons`` / ``devel``) directories. This is


should this just say "devel" instead of "addons / devel"? I think the contribution guide stated that addons was supposed to be stable as well.

I'm specifically looking under "Namespaces for Contributed and Experimental Code" in contribution_guide.rst.

We call it "mostly stable". I don't want to promise that addons is for sure stable.

doc/OnlineDocs/principles.rst

pyomo/addons/README.md

michaelbynum · 2025-11-03T23:55:08Z

pyomo/addons/README.md

+- Represent functionality that is **useful to the wider Pyomo community** but
+  not required for core operation.


I don't think this bullet adds any value, and I think it could be misconstrued.

michaelbynum · 2025-11-03T23:55:52Z

pyomo/addons/README.md

+Examples include:
+- `gdpopt`
+- `incidence_analysis`
+- `latex_printer`
+- `trustregion`
+- `viewer`


can't people just look to see what is in the addons directory? Why list it here?

I have an answer to this one! Because as of this PR, there will be nothing in the addons directory, so I wanted to list some actual examples while we go through the process of actually moving everything.

michaelbynum · 2025-11-03T23:58:23Z

pyomo/addons/README.md

+  - (PREFERABLE) A **reference page** in the Pyomo online documentation.
+  - (MINIMUM) A **README.md** file summarizing:
+    - Functionality
+    - Example usage or link to docs


I think "or link to docs" answers one of my earlier questions.

michaelbynum · 2025-11-03T23:59:52Z

pyomo/addons/README.md

+  - Declared in `setup.py` under the appropriate category.
+  - Publicly available on PyPI and/or Anaconda and reasonably maintained.
+
+### Backward Compatibility


Is this section a duplicate of "Stability and Maintenance" above?

michaelbynum · 2025-11-04T00:02:21Z

pyomo/devel/README.md

+Examples include:
+- `doe`
+- `piecewise`
+- `solver`
+- `sensitivity_toolbox`


Again, I don't think this list is needed.

michaelbynum · 2025-11-04T00:05:34Z

.gitignore

 *.egg-info/

 # Documentation builds
+doc/OnlineDocs/api


This is auto-generated stuff?

Yup - when you run make -C doc/OnlineDocs html, this is all the automatic api docs that get generated. Which are great, we love them, but I have almost accidentally committed them way too many times.

emma58

This looks nice--I put a lot of ideas in the comments.

doc/OnlineDocs/contribution_guide.rst

emma58 · 2025-11-04T18:50:50Z

doc/OnlineDocs/getting_started/installation.rst

+Optional Dependencies
+~~~~~~~~~~~~~~~~~~~~~

 Extensions to Pyomo, and many of the contributions in ``pyomo.contrib``,


Should pyomo.contrib be updated to be pyomo.addons and pyomo.devel here?

Not yet. We aren't moving anything out of contrib as of this particular PR. This file should be updated when we do actually move things.

doc/OnlineDocs/principles.rst

emma58 · 2025-11-04T19:01:22Z

doc/OnlineDocs/principles.rst

+* **Print statements:** Avoid printing directly to ``stdout``. Pyomo is a
+  library, not an application, and copious output can interfere with downstream
+  tools and workflows. Use the appropriate logger instead. Print
+  information only when the user has enabled or requested it.


Maybe we should add one about logging below this: We have settled on usually naming the logger using __name__, and I think we like that?

emma58 · 2025-11-04T19:02:22Z

doc/OnlineDocs/principles.rst

+* **Pull Request naming:** Pull Request titles are added to the CHANGELOG
+  and the release notes. The Pyomo development team reserves the right to
+  alter titles as appropriate to ensure they fit the look and feel of
+  other titles in the CHANGELOG.


@blnicho, didn't you have more specific guidance for this at one point too? Character limit and starting with "package:" or something like that that we fail to do most of the time?

emma58 · 2025-11-04T19:16:19Z

doc/OnlineDocs/principles.rst

+     #  This software is distributed under the 3-clause BSD License.
+     #  ___________________________________________________________________________
+
+  Update the year range as appropriate when modifying files.


A few more in-the-weeds ideas we could add to this list (or defer for a more in-the-weeds thing later--I'm not super attached to any of these, just brainstorming):

Loud failure is always better than silent failure: Write code to guard against cases you don't expect, even if you just raise NotImplementedErrors to start rather than handling them. The worst Pyomo bugs are the ones that silently do wrong or unexpected math.

Along this vein, if the behavior a user could reasonably expect is ambiguous, document assumptions well and fail loudly when they are violated.

Some Pyomo components are ActiveComponents and others are not (perhaps most importantly, Vars are not)... We define models to be the set of active components accessible from the root Block. This means many things, but in common gotchas, that m.component_data_objects(Var, ...) is almost always a bad idea: Usually you want get_vars_from_components with Constraint or (Constraint, Objective) as the ctype argument.

Writers and their ilk should always complain about active Pyomo components they do not recognize. (This is because Pyomo supports extended modeling environments such as DAE and GDP so it is easy to accidentally send unexpected structures to solvers expecting algebraic model formulations). There is a utility for this in pyomo.repn.util called categorize_valid_components.

We prefer not to use component names (or strings in general) to keep track of them in writers, algorithms, etc. pyomo.common.collections includes data structures where unhashable components (e.g., Vars) can be used as keys.

We prefer PEP8 naming conventions (descriptive names, snake case for functions, pascal case for classes, etc.) and also prefer functions be verb phrases and classes be noun-ish.

We have a deadly allergy to repeated (and especially copy-pasted) code--we're always happy to talk about design if something seems to be headed that way

Edit to the third bullet based on dev call discussion: If you want to gather all the variables on the model, it is recommended to use get_vars_from_components with (Constraint, Objective) as the ctype argument. This is as opposed to m.component_data_objects(Var, ...), which will get all the Vars declared on the model hierarchy, which is highly unlikely to be what is desired, as it has no mathematical meaning.

I strongly agree with all of these.

emma58 · 2025-11-04T19:21:55Z

pyomo/devel/example/plugins/ex_plugin.py

 @TransformationFactory.register(
-    'contrib.example.xfrm',
-    doc="An example of a transformation in a pyomo.contrib package",
+    'devel.example.xfrm', doc="An example of a transformation in a pyomo.devel package"
 )
 class Xfrm_PyomoTransformation(Transformation):
    def __init__(self):


Perhaps while we're making this change, we should adjust naming here? I'd suggest devel.example.transform with a class name of just ExamplePyomoTransformation or something. I also think it would be good to not have a plugins directory in example and just flatten it out like we wish we had so many other places...

doc/OnlineDocs/contribution_guide.rst

jsiirola · 2025-11-12T08:18:40Z

doc/OnlineDocs/contribution_guide.rst

+
+As a result, Pyomo is transitioning to a more structured contribution
+model with two clear namespaces:
+


Suggested change

* ``pyomo`` core - These modules form the foundation of the Pyomo environment,

including the base expression systems, modeling components, model compilers,

and solver interfaces. The core development team has committed to

maintaining these capabilities, adhering to the strictest policies for

testing and backwards compatibility.

I agree with the words but not the placement here. I put them above so it doesn't confuse the message here, which is primarily focused on, "This is how we are splitting pyomo.contrib."

See https://github.com/Pyomo/pyomo/pull/3768/files#diff-c20ba9eb661148c0c7e8bd2e58c5b0fb13ad34defb7f8bd18e6039c17308a482R424

doc/OnlineDocs/contribution_guide.rst

doc/OnlineDocs/principles.rst

jsiirola · 2025-11-12T18:07:14Z

doc/OnlineDocs/principles.rst

+* **File headers:** Every ``.py`` file must begin with the standard Pyomo
+  copyright header:
+
+  .. code-block:: text


Can this be .. literalinclude::'ed from pyomo/__init__.py (that way the documentation is kept in sync with the codebase)?

jsiirola · 2025-11-12T18:08:57Z

doc/OnlineDocs/principles.rst

+     #  This software is distributed under the 3-clause BSD License.
+     #  ___________________________________________________________________________
+
+  Update the year range as appropriate when modifying files.


jsiirola · 2025-11-12T18:11:32Z

pyomo/devel/example/__init__.py

-from pyomo.contrib.example.foo import *
-from pyomo.contrib.example import bar
+from pyomo.devel.example.foo import a
+from pyomo.devel.example import bar


I would like to propose removing contrib/example entirely and replacing it with a section in the Online Docs

mrmundt · 2026-01-20T23:20:50Z

doc/OnlineDocs/contribution_guide.rst

+  prototypes, or specialized modeling components. Functionality under
+  this namespace may change or be removed between releases without
+  deprecation warnings.
+


Reminder for myself to add in pyomo.unsupported as another new unique namespace.

Introduce formal principle documentation to support contrib changes

74aee3f

mrmundt requested review from blnicho, emma58, jsiirola and michaelbynum October 21, 2025 23:03

Found a weird phrasing that I missed when proofreading

ca02f4e

michaelbynum reviewed Oct 22, 2025

View reviewed changes

doc/OnlineDocs/principles.rst Outdated Show resolved Hide resolved

mrmundt added 3 commits October 22, 2025 08:22

Whoops, no init files

8e18ee1

Move redundant information and do a big rework of contribution guide

4822e6d

Typo

9e4b67c

mrmundt requested a review from michaelbynum October 22, 2025 15:43

blnicho and others added 3 commits October 27, 2025 07:05

Merge branch 'main' into addons-devel

5f37e4a

Wrong word in contribution guide

1be9b29

Minor corrections in installation docs

5d7f102

blnicho reviewed Oct 27, 2025

View reviewed changes

Address blnicho's comments

ed7f9dd

mrmundt requested a review from blnicho October 28, 2025 14:43

blnicho approved these changes Oct 28, 2025

View reviewed changes

mrmundt and others added 3 commits October 28, 2025 13:47

Merge branch 'main' into addons-devel

6999a91

Add convention about printing to stdout

cff39eb

Merge branch 'main' into addons-devel

1b023a8

michaelbynum reviewed Nov 3, 2025

View reviewed changes

doc/OnlineDocs/contribution_guide.rst Outdated Show resolved Hide resolved

doc/OnlineDocs/contribution_guide.rst Outdated Show resolved Hide resolved

michaelbynum approved these changes Nov 4, 2025

View reviewed changes

Merge branch 'main' into addons-devel

ce089c4

emma58 reviewed Nov 4, 2025

View reviewed changes

blnicho added this to Pyomo 6.10 Nov 11, 2025

github-project-automation bot moved this to Todo in Pyomo 6.10 Nov 11, 2025

blnicho moved this from Todo to Review In Progress in Pyomo 6.10 Nov 11, 2025

jsiirola reviewed Nov 12, 2025

View reviewed changes

pyomo-autotest added the AT: STALE label Nov 24, 2025

mrmundt commented Jan 20, 2026

View reviewed changes

Merge branch 'main' into addons-devel

af64211

pyomo-autotest removed the AT: STALE label Jan 21, 2026

mrmundt and others added 5 commits January 21, 2026 14:36

Merge branch 'main' into addons-devel

9c03e5a

Save state: Partially addressed comments

f12f1f7

Merge branch 'main' into addons-devel

31698dc

Update READMEs to have them point to official guidelines

8feac13

Missing end-bold-note

92b607c

		change until the next major Pyomo release, which will coincide with a
		new edition of the book.

		- Represent functionality that is useful to the wider Pyomo community but
		not required for core operation.


		As a result, Pyomo is transitioning to a more structured contribution
		model with two clear namespaces:

+* ``pyomo`` core - These modules form the foundation of the Pyomo environment,
+  including the base expression systems, modeling components, model compilers,
+  and solver interfaces.  The core development team has committed to
+  maintaining these capabilities, adhering to the strictest policies for
+  testing and backwards compatibility.

Introduce formal principle documentation to support contrib changes #3768

Are you sure you want to change the base?

Introduce formal principle documentation to support contrib changes #3768

Conversation

mrmundt commented Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Fixes None but starts to address #2460

Summary/Motivation:

Changes proposed in this PR:

Legal Acknowledgement

Uh oh!

codecov bot commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

blnicho left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

michaelbynum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

michaelbynum left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emma58 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mrmundt commented Oct 21, 2025 •

edited

Loading

codecov bot commented Oct 22, 2025 •

edited

Loading

emma58 Nov 4, 2025 •

edited

Loading