Links with special characters do not work anymore

[privefl]

My bookdown is available here.

All the sections/links with a special character (like é or î) do not work anymore when I click on them.
It used to work very recently.
I have updated a bunch of packages since.
It does not seem to come from the bookdown version somehow, because I tried some older version.

[cderv]

I believe this could be related to Pandoc version. Can you check if you have recently updated?

Pandoc 3.2.1 could be the problem.

Possibly relates to this:

• href are escaped by Pandoc 3.2.1 breaking toc rmarkdown#2580

We've just discuss it with @yihui and should be fixed in next rmarkdown version.

In the meantime, using an older pandoc version can help - and using pandoc package (https://cran.r-project.org/package=pandoc) can help do the test.

Can you confirm your pandoc version ? And downgrading help ?

[privefl]

packageVersion("pandoc") says I don't have pandoc installed
The CRAN version is 0.2.0; I guess you're talking about something else than the R package?

[privefl]

It seems I didn't have pandoc installed?
I tried with all of these: "3.7.0.2" "3.2" "3.1.13" , and it didn't change the problem.

[cderv]

Sorry for the wording here. I shouldn't have assumed you knew this stuff. Let me clarify.

It used to work very recently.
(...)
It does not seem to come from the bookdown version somehow, because I tried some older version.

This is a hint that the problem could be some updates of Pandoc binaries in the workflow that builds your book. Is your book built on CI? Did you check Pandoc CLI binary version there ? (I am not talking of the R package here).
Also can you reproduce locally ? Which version are you using ? rmarkdown::find_pandoc() should tell you which version bookdown will use. You may have updated without knowing if (1) you don't install pandoc system-wise (2) but rely on version shipped with IDE like RStudio or Positron. An update to an IDE could lead to an update in Pandoc.

and using pandoc package (cran.r-project.org/package=pandoc) can help do the test.

The pandoc R package is documented here: https://cderv.github.io/pandoc/index.html
Reading this should clarify its usage. It will help installing several pandoc versions and managing, which is useful mainly for test. Then using pandoc::pandoc_activate() should also setup the version for use with rmarkdown. Calling rmarkdown::find_pandoc() after activation can check this.

Is your book available as source on github so that I can clone and try investigate more closely to see if this is indeed the same as rstudio/rmarkdown#2580 ?

Thank you

[privefl]

Yes, I get the same issue locally. The source code is here.

> rmarkdown::find_pandoc()
$version
[1] ‘3.4’

$dir
[1] "C:/Program Files/RStudio/resources/app/bin/quarto/bin/tools"

> pandoc::pandoc_installed_versions()
[1] "3.7.0.2" "3.2"     "3.1.13"   # the versions I tried

> pandoc::pandoc_activate("3.2"); rmarkdown::find_pandoc()
✔ Version '3.2' is now the active one.
ℹ Pandoc version also activated for rmarkdown functions.
$version
[1] ‘3.2’

$dir
[1] "C:/Users/flori/AppData/Local/r-pandoc/r-pandoc/3.2"

Thank you

[privefl]

I don't have this problem with bs4_book instead of gitbook.

[cderv]

This is because this is a change in Pandoc impacting HTML4 output only, and bs4_book() produces HTML5 outputs.

[cderv]

So I rendered your book with Pandoc 3.6.3

> rmarkdown::find_pandoc()
$version
[1] ‘3.6.3’

$dir
[1] "C:/Users/chris/scoop/apps/rstudio-daily/current/resources/app/bin/quarto/bin/tools"

And looking at the problem it is that the TOC link are not linking to page
For example http://127.0.0.1:4321/#pr%C3%A9vention-des-blessures-1 links to an id.

Links inside pages, for linking to header are working like

So something is off in how the TOC is built for gitbook

[privefl]

What can I do about it?

[cderv]

So the problem is inside restore_links

bookdown/R/html.R

Lines 959 to 978 in 52c8724

	restore_links = function(segment, full, lines, filenames) {
	# if there is only one chapter in the HTML in total, no need to restore links
	if (length(lines) <= 1) return(segment)
	r = '<a href="#([^"]+)"'
	m = gregexpr(r, segment)
	regmatches(segment, m) = lapply(regmatches(segment, m), function(x) {
	if (length(x) == 0) return(x)
	links = gsub(r, '\\1', x)
	for (i in seq_along(links)) {
	a = grep(sprintf(' id="%s"', links[i]), full, fixed = TRUE)
	if (length(a) == 0) next
	a = a[1]
	x[i] = sprintf(
	'<a href="%s#%s"', filenames[which.max(lines[lines <= a])], links[i]
	)
	}
	x
	})
	segment
	}

We do retrieve ID #<id> in TOC part in the <a href=, and try to find the split files that has the id=<id>

Problem is that now:

• #<id> is url encoded by Pandoc.

   > x
   [1] "<a href=\"#les-entra%C3%AEneurs\"

• but the id = in the document section won't be

   > grep("id=\"les-entra", full , value = TRUE)
   [1] "<div id=\"les-entraîneurs\" class=\"section level2 unnumbered hasAnchor\">"

So we need to account for this while restoring the links for split files.

here

bookdown/R/html.R

Line 968 in 52c8724

a = grep(sprintf(' id="%s"', links[i]), full, fixed = TRUE)

we do get

> grep(sprintf(' id="%s"', links[i]), full, fixed = TRUE)
integer(0)

but if we decode

> grep(sprintf(' id="%s"', URLdecode(links[i])), full, fixed = TRUE)
[1] 14

And then when adding the filename

bookdown/R/html.R

Lines 971 to 973 in 52c8724

	x[i] = sprintf(
	'<a href="%s#%s"', filenames[which.max(lines[lines <= a])], links[i]
	)

this will be correct

> sprintf(
+     '<a href="%s#%s"', filenames[which.max(lines[lines <= a])], links[i]
+ )
[1] "<a href=\"index.html#les-entra%C3%AEneurs\""

@yihui how do you want to handle that ? Should we just decode always using URLdecode(links[i]) ? Or conditionnaly to Pandoc version ?

I don't think there is any harm in trying to decode always... 🤔

This won't be easy to test between version... but we may be able to check the output file after processing.

[cderv]

I confirm that this fix the issue in @privefl book (thanks a lot for the reproducible book example !! so valuable !!)

diff --git a/R/html.R b/R/html.R
index 5739ca10..09948b69 100644
--- a/R/html.R
+++ b/R/html.R
@@ -965,7 +965,7 @@ restore_links = function(segment, full, lines, filenames) {
     if (length(x) == 0) return(x)
     links = gsub(r, '\\1', x)
     for (i in seq_along(links)) {
-      a = grep(sprintf(' id="%s"', links[i]), full, fixed = TRUE)
+      a = grep(sprintf(' id="%s"', URLdecode(links[i])), full, fixed = TRUE)
       if (length(a) == 0) next
       a = a[1]
       x[i] = sprintf(

Testing with Pandoc 3.1.7, I can notice that

> x
[1] "<a href=\"#les-entraîneurs\""

we don't get the escaping (so that was really the change)

then url decoding does not modify

> URLdecode(links[i])
[1] "les-entraîneurs"

and filename is correctly found

> sprintf(
+     '<a href="%s#%s"', filenames[which.max(lines[lines <= a])], links[i]
+ )
[1] "<a href=\"index.html#les-entraîneurs\""

so it seems ok...