Migration to sphinx-gallery #8247

Closed
wants to merge 21 commits into
from

Conversation

Contributor

NelleV commented Mar 9, 2017 • edited

This work is still in progress.

Right now, the documentation builds two gallery. The old one, which stays identical as on current master and a sphinx-gallery one. The sphinx-gallery one needs specific examples formatting, hence only includes so far a subset of our example.

I'll slowly update this PR with progress and build this on a local website to show how it looks. Once we decide it is "good enough", the switch can easily be done through a URL switch.

Here is the sphinx-gallery build

NelleV added the Documentation label Mar 9, 2017

NelleV added the MEP12 label Mar 11, 2017

NelleV changed the title from [WIP] Migration to sphinx-gallery to [MRG] Migration to sphinx-gallery Mar 23, 2017

tacaswell changed the title from [MRG] Migration to sphinx-gallery to [MRG+1] Migration to sphinx-gallery Mar 23, 2017

@tacaswell

I still have reservations about needing to re-name all of the examples, but the benefits out weigh the cost.

@@ -12,3 +12,4 @@ ipython
mock
colorspacious
pillow
+sphinx-gallery
@phobson

phobson Mar 24, 2017

Member

we need to add graphviz to this list

@NelleV

NelleV Mar 24, 2017

Contributor

unfortunately, we can't as this file is meant to be used by pip, and graphviz is not pip installable.

QuLogic added this to the 2.1 (next point release) milestone Mar 24, 2017

@@ -30,7 +30,7 @@ if [[ $BUILD_DOCS == false ]]; then
fi
else
cd doc
- python make.py html -n 2
+ python make.py html
@QuLogic

QuLogic Mar 24, 2017

Member

Does parallel build not work?

@NelleV

NelleV Mar 25, 2017

Contributor

No, but ironically, the parallel build doesn't change the run time :D
We can easily activate the flags in sphinx-gallery to make it parallel-safe, but considering there is no gain at all, I haven't created the PR yet.
Actually making the code parallel requires more work, but should be doable.

@tacaswell

tacaswell Apr 9, 2017

Owner

I I have definitely seen good speed up on building the docs (as they are now) but have found you need to use ~2x the number of workers compared to the number of core.

doc/conf.py
@@ -54,6 +55,12 @@
"numpydoc to build the documentation.")
try:
+ import sphinx_gallery
+except ImportError:
+ raise ImportError("No module named sphinx-gallery - you need to install "
@QuLogic

QuLogic Mar 24, 2017

Member

The module is called sphinx_gallery.

doc/conf.py
+ import sphinx_gallery
+except ImportError:
+ raise ImportError("No module named sphinx-gallery - you need to install "
+ "numpydoc to build the documentation.")
@QuLogic

QuLogic Mar 24, 2017

Member

Need to install sphinx-gallery, not numpydoc.

+# Sphinx gallery configuration
+sphinx_gallery_conf = {
+ # path to your examples scripts
+ 'examples_dirs' : '../examples',
@QuLogic

QuLogic Mar 24, 2017

Member

Does this mean we can delete the mpl_examples symlink?

@NelleV

NelleV Mar 25, 2017

Contributor

Not yet. Our plot directive writes into mpl_examples, and unfortunately we have a bit more work to do before getting rid of this (including more examples to migrate). This will be my next step.

+ # path to your examples scripts
+ 'examples_dirs' : '../examples',
+ # path where to save gallery generated examples
+ 'gallery_dirs' : 'auto_examples'}
@QuLogic

QuLogic Mar 24, 2017

Member

Can we not put it in gallery instead? Or if it were mpl_examples, then some of the lines that were changed would not have been necessary.

Whatever it becomes, I don't like the auto_ prefix; it's an internal concern and doesn't really make any sense to present that in the URL to a user from a semantic point of view.

@NelleV

NelleV Mar 25, 2017

Contributor

It's the default and is a convention followed by many (if not all) the projects. I'd rather stick with it for consistency.
We can't rename this to mpl_examples as mpl_examples still exists.

@QuLogic

QuLogic Mar 26, 2017

Member

I'm not sure those are compelling reasons. I went back through the blame 7 years and couldn't find any reasoning behind it (that's not to say that there wasn't one.) My reasoning here is that URLs should be discoverable, simple, and at least try to give some semantic meaning.

A user might go to matplotlib.org/gallery or matplotlib.org/examples directly (yes, there's Google, but let's assume they're not using it for some reason), but there's a 99% chance that no-one will spontaneously go to matplotlib.org/auto_examples.

And what does the "auto_" prefix mean? It means that the rst was automatically generated from the example Python code. This is a meaningless distinction to a user. When they see "auto", they see "automatically generated" aka "not hand-written" aka "not very good". I'm not going to claim that all of our examples are 100% clear, but they've all been hand-written in some fashion, and some of them are even quite amazing.

In fact, for that same reasoning, I've just talked myself into retracting the "mpl_examples" suggestion, since the "mpl_" prefix is redundant and meaningless in this context. Also, our existing site uses matplotlib.org/examples/ and matplotlib.org/gallery.html; I don't think we should break that.

@NelleV

NelleV Mar 26, 2017

Contributor

For the time being, the folder cannot be neither "examples" nor "mpl_examples", and this will not change in the foreseeable future considering the state of our examples (unless we remove all examples that are not SG compatible).

@dopplershift

dopplershift Mar 27, 2017

Contributor

It's not possible to rename the current examples/mpl_examples to make room for the gallery-based examples? I strongly dislike the auto_examples name as well--I've overridden that name for sphinx-gallery in my own projects.

@NelleV

NelleV Mar 28, 2017

Contributor

It is possible, but (1) so far there has been no compelling reason to do it; (2) the amount of work required to change this is sufficiently large that I am not doing it.

To extend on point 1, our naming patterns for example plots and links is obscure, inconsistent and so poorly descriptive of what is in the examples that I really don't think the name of the folder is ever going to help or prevent our users from finding anything (we have about 5 examples named "demo bar_X", with X an incremental number). We also need to be realistic: no one is ever going to guess our url or even attempt to guess our url, and possibly not even look at the url.
Second, it is a convention that is followed by most projects and following conventions have many benefits for easing contribution, maintenance, etc.
Third, taking personal preferences into account is not for me a compelling reason (I personally prefer auto_examples over both examples and mpl_examples, as it makes it to a certain extent more clear that this folder is auto-generated and thus should not be modified by our contributors, and this is typically the kind of questions where if you have N developers in the room, you'll have N opinions).

@tacaswell

tacaswell Apr 9, 2017

Owner

Will this end up exposed in the URL?

If so, I think we should pick something either a) shorter (ex sg or ae) or b) something more meaningful.

@choldgraf

choldgraf Apr 9, 2017

Contributor

I does show up in the URL, for example this is our gallery at MNE-python. It does have auto-examples in the page, though that's sorta become the convention with sphinx-gallery (same goes for scikit-learn: http://scikit-learn.org/stable/auto_examples/index.html).

I seem to remember conversations with folks that this was a non-trivial thing to change but I could be wrong...it's been working out well enough for us. If you want you could use an .rst file that contains the gallery itself, then the objects that the gallery links to would be in the auto_examples folder.

@dopplershift

dopplershift Apr 9, 2017

Contributor

Nope, it's completely trivial to change: https://unidata.github.io/MetPy/examples/index.html

I'm going to argue one more time NOT to use auto_examples. Whether they're generated automatically or not is immaterial to the user, so we should NOT be bleeding that implementation detail of our documentation build procedure to the outside world.

@NelleV

NelleV Apr 10, 2017

Contributor

It's not that trivial, because matplotlib somehow does I don't know what in the plot directive that makes this a pain in the butt to change. Which is why I asked for feedback a month ago.

@choldgraf

choldgraf Apr 10, 2017

Contributor

FWIW I'll give this a shot to change the name to examples_gallery. It doesn't look completely trivial in the sense that there are links hard-coded all over the codebase. There are also some weird errors happening inside the plot directive that I can try to debug. I'll open a separate PR then if I can figure it out.

+ # path where to save gallery generated examples
+ 'gallery_dirs' : 'auto_examples'}
+
+plot_gallery = True
@QuLogic

QuLogic Mar 24, 2017

Member

Blank line after.

Member

QuLogic commented Mar 24, 2017

Is there a way to make sphinx-gallery less verbose? It prints out that it's going to run an example, then that it ran the example and how long it took plus a blank line, then at the end, it prints out how long it took to run all the examples again. That's 4 lines for each example.

Contributor

NelleV commented Mar 25, 2017

I don't think there is a way to make sphinx-gallery less verbose. It also doesn't bother me: it's actually great for debugging.

NelleV added some commits Mar 8, 2017

@NelleV NelleV DOC starting the migration to sphinx
- Added sphinx-gallery to doc/conf.y
- Renamed all conflicting README.txt to README in our example folders
b918016
@NelleV NelleV DOC Set up sphinx-gallery in parallel of our own gallery b110b1f
@NelleV NelleV DOC added sphinx-gallery as a requirement ba6d268
@NelleV NelleV DOC Added readme files to include examples in SG c908f98
@NelleV NelleV DOC SG doesn't support parallel builds yet c61c856
@NelleV NelleV DOC lines, bars and markers is now full SG eaaf182
@NelleV NelleV DOC examples/statistics is fully SG 4f92501
@NelleV NelleV DOC pie and polar charts is now full SG 40eab47
@NelleV NelleV DOC subplots, axes and figures are now full SG 9738124
@NelleV NelleV DOC added ticks and spines to sphinx-gallery 8f8d316
@NelleV NelleV DOC image, contours and fields are fully SG compatible 55a5840
@NelleV NelleV DOC showcase are fully SG compatible 9bf91a8
@NelleV NelleV DOC text, labels and annotations are fully SG compatible f50ab00
@NelleV NelleV DOC tick and spines is now full SG compatible 7c79b12
@NelleV NelleV DOC shapes and collections is now fully SG compatible 1ed25aa
@NelleV NelleV DOC mplot3d is fully SG-compatible 8bc4597
@NelleV NelleV DOC The gallery is now built with sphinx-gallery
:D
d5583f8
@NelleV NelleV DOC all plots in the gallery now produce images 8f5f185
@NelleV NelleV DOC API is now fully SG-compatible 9d805a1
@NelleV NelleV DOC linking to auto_examples when possible 4a13444
@NelleV NelleV DOC FIX nitpicks in the conf file
8b764e7
Contributor

choldgraf commented Apr 8, 2017

Looking forward to seeing this get merged into the website! Once it's merged I can spend a little time trying to create more narrative-driven examples out of the ones that are there and generally improving them. Is there anything holding this back right now?

@@ -75,7 +75,8 @@
<p>Matplotlib tries to make easy things easy and hard things possible.
You can generate plots, histograms, power spectra, bar charts,
errorcharts, scatterplots, etc., with just a few lines of code.
- For a sampling, see the <a href="{{ pathto('users/screenshots') }}">screenshots</a>, <a href="{{ pathto('gallery') }}">thumbnail</a> gallery, and
+ For a sampling, see the <a href="{{ pathto('users/screenshots')
+ }}">screenshots</a>, <a href="{{ pathto('auto_examples/index') }}">thumbnail</a> gallery, and
@tacaswell

tacaswell Apr 9, 2017

Owner

I assume the linebreak does not affect the templating?

@NelleV

NelleV Apr 9, 2017

Contributor

No, it works fine

NelleV closed this Apr 10, 2017

Contributor

NelleV commented Apr 10, 2017

Chris is taking over this work. I don't have time and the energy to continue working on matplotlib's migration to sphinx-gallery.

dstansby added the Duplicate label Apr 12, 2017

QuLogic changed the title from [MRG+1] Migration to sphinx-gallery to Migration to sphinx-gallery Apr 30, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment