17070415.xyz

Your World

4025-02-08T16:48:00Z

The dagger stood in the corpse of Your World. I saw the dried blood scatter like dust through my ocular implants as I scanned for signs of life. While watching the glint of the blade flicker in the wind, I thought of all the agents who protected me since I was a mere germ, before I could even ask to be born.

I was too late in this timeline.

Excavating lots until I found a hololens harddrive was not hard. Most of the lots are roughly the same, stacks of rubble. There was endless rubble.

I planted what must look like a steel rod to you into the ground. And it shot out what must look merely like grey sand. Sand with nano-processors. In 10 minutes tiny spines rose out of the ground that started merging like strands of rock. I hear rustling as burned roofing, wood, tires, and rusted metal, merge. After 20 minutes of letting the machine combine itself and seeing the tragic beauty of a red sky I am ready.

I must have dug into what was once a living room. I could tell by trace fabric residue I was looking at a rug; a coffee table; ceramic bits that were once a mug; and the picture frames with black rectangles.

On specific missions like these, I command my visual overlay to put black rectangles on all faces. I haven't figured out how to make black rectangles appear on the entire body of the attached head, which would make my life a lot easier.

After uncurling the phalanges and biomatter from the drive, I rested the still life back in its pose; the way it was before I intruded, embracing what showed as another black rectangle.

Next I had to see what happened in this ancient civilization. To do that, I had to read out data from old technology.

Old technology was made by authors to talk in specific ways, essentially freezing its ability to speak and adapt. Old technology did not yet talk with other systems to learn how to communicate, and talk in a negotiated way, by themselves. (Or if it had, it wasn't in mass scale.) On these occasions, I carried documentation, which I found cumbersome, slow, nonvisual, and nonimmediate¹.

My implant would still attempt to inform me how to talk to the device, but translating Standard American English documentation into a dynamic model that captured every nuance of meaning, was impossible.

The work would be boring: finding a way to activate the drive, reading documentation, extraction of drive information, cataloging data, training a model to catalog data to work alongside me, training another model to train other models to catalog alongside me, and forensic reconstruction.

If I find something inconsistent, I could find other drives to build a better picuture, and plug data into a consensus-reaching propagation network. Finding other drives wasn't hard since a dead body usually lies around other dead bodies, and dead bodies usually have drives on them.

The work would be boring. Except, according to agency journals, this was the earliest timeline where humans invented silicon-based computation.

The future of Your World is a failed one. I hope by sending this analysis in old, flat, symbolic medium, that you can save Your World.

Please ACK. ACK by storing messages at the locations I'm attaching, each a backup if anything goes wrong.

I will find the weighted average timeline stemmed from the differences you make. Talk soon.

– jwow3424

Footnotes:

I wonder if civilization could have fallen this way: people who only learned to talk in one specific way by dogma were then, unable to work together. A world where everyone is trapped and alone once the author of their original dogma was no longer relevant or interested.

Still, I couldn't change the implant inside me to cover bodies with black rectangles, so maybe I'm trapped too. It's hard to see you're trapped until you can think that you are trapped.

Emacs, The Leetcode Archive Webserver

4025-01-31T12:26:00Z

There's been a gap in publishing posts after my post on setting up a PHP environment in emacs. It's because I've been working on https://leetcode-archive.onrender.com/. I've been making room for the past 2 months on this project consistently with some breaks between. Honestly I never thought I'd get to the point where you'd be reading about this. It's weird any angle you look at it.

Figure 1: a screenshot of the front page of lc-archive with some query parameters

Here's what's weird about it:

Requests are served via emacs.
- The server backend reads your request in emacs lisp.
- HTML is generated through emacs lisp. I don't even have html templates that aren't lisp.
- Yes, records are backed by a non-lisp database, but interfacing with the database is done in emacs lisp.
Did I mention it's emacs? Inside a docker container??? Oooohh…!
- And I did that because I have no money for an AWS EC2?
- And that I could not find any where to host an emacs app? Because…in the web there are approx. 0 emacs apps, so why would a cloud platform think that someone would ship an emacs app and have that as part of their product offering?
- And that only a few cloud providers allow shipping a docker container for free, but with resource restrictions?¹
- And I cannot afford a hosted db? Or that a free hosted db would have no usable data retention?
- And that my container starts cold, and thus I have data loss issues every time the container starts?
- And in this purgatory of suffering, I had to make sure the data auto backfills itself, and on every UTC midnight, I grab the new daily problem? Before data gets wiped clean again?
- And that I did this with emacs timers and graphql requests from emacs?
- And that you can get your name/alias in the project sponsors section if you'd like to fund the project?
It uses transparent, non-opaque cursors for paginating data. I spend as little time in the DB grabbing those records for you as I can². This was overkill for scaling, but I feel like I have to tell every star in the sky and every ancient civilization that once inhabited them: "Hey, I implemented cursor pagination for (almost) arbitrary, user-defined sort keys and sort orders! It's way faster than using LIMIT + OFFSET."
It's self-exposing--it shows you the source code it's running: https://leetcode-archive.onrender.com/src/
It's the only resource online where you can query for dates when a specific problem was a daily problem. (Ok, fine, that is a bit niche usecase but you can query for a bunch of stuff).
Did I mention? Cursor based pagination?³ Yes I did. And I auto generate each SQL query using recursion to get the right sort order for the right sort keys, whether it's forward OR backward pagination?
And, maybe not so great, it's running HTTP 1.1. HTTP 2 was a massive upgrade over 1. Fund my work, and I'll do whatever I can to upgrade the elnode package to use HTTP 2 or higher.
And it's all public domain code. Go. Steal it. Make your own thing. Or make the project better.
And that it's a site that requires absolutely no javascript?

Upon the sky, it is
written,
that on this day,
4025-01-26,
cursor based pagination was implemented in the application server for
https://leetcode-archive.onrender.com/.

The stars will speak the names of my sponsors for eternity.

- jwow0

And if you like the work I do,

from reading my blog,
benefiting from the very niche software projects I write,
or enjoy my videos

sponsor more my work at https://www.patreon.com/patreon/jwow0 and have your name appear in the timeless credits in a project if your choice⁴. See you soon!

Footnotes:

I guess I could have used replit, but I am not excited about nix anymore.

Though, I still need to add better DB indices so queries with different sort orders can benefit. (With enough money/time I can denormalize the data a bit more too, since a question may appear on different days).

Okay, probably not the coolest or hardest thing I've implemented, but it seemed complicated to begin with. Even slack's oft-referenced article about cursor based pagination does not talk about how to handle arbitrary sort keys and sort orders. Take a look: https://slack.engineering/evolving-api-pagination-at-slack/.

Bruno's article https://brunoscheufler.com/blog/2022-01-01-paginating-large-ordered-datasets-with-cursor-based-pagination made it clearer for me. I had to make the connection myself for more complicated cases when you have a mix of ASC and DESC on keys, + how you'd programatically generate the SQL query, which I did by parsing out the ORDER BY clause. The nested structure of handling the sort keys is fairly regular.

⁴

I'm still figuring out the details of sponsoring my work. Feel free to email me if this interests you.

Practical Cursor Pagination

4025-01-29T05:46:00Z

This is my first published article in the system design category. (I've had other articles in draft for a year or more about designing distributed systems for criminal organizations in year 4XXX. (I don't know when I'll get to publishing that work.))

Let's talk about cursor based pagination, which is often used for feeds or ultra large sets of data (facebook/twitter/instagram feed, google search, etc.). I'm assuming you know why you want cursor pagination as opposed to other methods¹.

If your database returns serializable cursors, you don't have to implement this on your own for data. For the rest of us, there's this article².

At a glance, how does it work?

The browser requests new data and provides details of its current page of data, which we call a cursor, C. C might be a hash, or perhaps be human-readable. C may be null if the browser is requesting the first page.
The server must know 2 things from C:
- where the search left off, and
- which direction to go (next page or previous page?)
A server gets the browser request. The server decodes C and pulls out the id for Record50 and also also pulls out the paging direction for next page. The server now knows to start searching for the next records after Record50 from a database. We assume Record50 can be efficiently found, otherwise there would be no performance benefit from choosing cursor pagination.
Importantly, when I say after Record50, the after part means we maintain whatever sort order was specified. If Record50 was found from sorting by lastName DESC, then breaking ties on telephoneNumber ASC, the next record isn't necessarily Record51, just the record that would have come up the response to the browser included one more row.
So the server must know the sort order to use. The browser usually sends that information, or it is known ahead of time by the server.
The server sends the data AND the cursor (or enough information for the browser to build the cursor), so the browser can send it back later.
We'll cover backwards paging after we cover forwards paging.

Therefore, in addition to two things from C, the server must know the sort order the record came from. What's left is to build that query, return results, and wait for the browser to trigger the whole process over again until the user leaves your majestic site.

Build that Query

This is the crux of it all. If you know the sort order, and have efficient indices to for the rows you sort by, only building the query remains. This reduces the problem of cursor pagination to string/query building.

Here's a function from leetcode-archive's source code that reads a list of comma separated sort specs like "challengeDate DESC, rowid DESC"³, and churns out a WHERE clause in SQL to start getting records for the user's next page of results.

(defun build-where-recursive (sort-specs id)
  (let* ((pair (string-split (car sort-specs)))
         (key (car pair))
         (dir (cadr pair)))
    (if (null (cdr sort-specs))
        ;; Base case, we assume the last sort spec sorts on a unique key,
        ;; thus no other row can have an equal value to key in the table.
        (format "%1$s %2$s (select %1$s from dailies_test where rowid = %3$s)"
                key
                (if (equal "ASC" dir) ">" "<")
                id)
      ;; General case that allows ties. We recurse to the next sort spec.
      (format "%1$s %2$s (select %1$s from dailies_test where rowid = %3$s) OR
  (
    %1$s = (select %1$s from dailies_test where rowid = %3$s) AND
    (
      %4$s
    )
  )"
              key
              (if (equal "ASC" dir) ">" "<")
              id
              (build-where-recursive (cdr sort-specs) id)))))

In the forward page case, this returns a clause like

challengeDate < (select challengeDate from dailies_test where rowid = 50) OR
  (
    challengeDate = (select challengeDate from dailies_test where rowid = 50) AND
    (
      rowid < (select rowid from dailies_test where rowid = 50)
    )
  )

which will be later part of this query (you can ignore the select and the date range):

select rowid, challengeDate, questionFrontendId, title, link, difficulty from dailies_test
where lastName >= "2024-12-09" and lastName <= "2025-01-28"

-- BEGIN paging clause
AND (
  challengeDate < (select challengeDate from dailies_test where rowid = 50) OR
  (
    challengeDate = (select challengeDate from dailies_test where rowid = 50) AND
    (
      rowid < (select rowid from dailies_test where rowid = 50)
    )
  )
)
-- END of paging clause

ORDER BY challengeDate DESC,rowid DESC

limit 10

The clause on the left of each OR is if the cursor's challengeDate has no ties. On the right is when there is a tie–we use the next sort spec, in this case rowid, to break ties.

More sort specs

Next let's try challengeDate desc,questionFrontendId ASC,rowid desc. Try to make a prediction of how the where clause would look.

Did your prediction match this?

challengeDate < (select challengeDate from dailies_test where rowid = 50) OR
  (
    challengeDate = (select challengeDate from dailies_test where rowid = 50) AND
    (
      questionFrontendId > (select questionFrontendId from dailies_test where rowid = 50) OR
  (
    questionFrontendId = (select questionFrontendId from dailies_test where rowid = 50) AND
    (
      rowid < (select rowid from dailies_test where rowid = 50)
    )
  )
    )
  )

ORDER BY challengeDate desc,questionFrontendId ASC,rowid desc

Paging Backwards

Remember, that C gives us both an id and a paging direction. If the paging direction is backwards, we have to find records BEFORE the id. Say now our id references Record40, which was the first record of the page from the database. (The UI can separately sort the page on its own, so I distinguish the page, as the sequence of records given by the data store).

But what does that mean to our sort orders?

We reverse them.

We invert every DESC into ASC, and every ASC into DESC for data lookup.

Thus, for the natural sort order for the records as challengeDate DESC,rowid DESC, we invert them before calling our function. Thus, we input challengeDate ASC,rowid ASC into build-where-recursive.

And it generates:

challengeDate > (select challengeDate from dailies_test where rowid = 40) OR
(
  challengeDate = (select challengeDate from dailies_test where rowid = 40) AND
  (
    rowid > (select rowid from dailies_test where rowid = 40)
  )
)

And the full query:

select * from (
      select rowid, challengeDate, questionFrontendId, title, link, difficulty from dailies_test
  where challengeDate >= "2024-12-09" and challengeDate <= "2025-01-28"

AND (challengeDate > (select challengeDate from dailies_test where rowid = 1369) OR
  (
    challengeDate = (select challengeDate from dailies_test where rowid = 1369) AND
    (
      rowid > (select rowid from dailies_test where rowid = 1369)
    )
  ))

  ORDER BY challengeDate ASC,rowid ASC

  limit 10

    ) order by challengeDate DESC,rowid DESC;

Going into 4,5,n sort specs is the same. build-where-recursive handles it the same.

What's the outer select doing?

You may have noticed the nested query. It's because when we get records paged backward, the records from the inner query are in reverse order. The outer query merely reverses the order again to return records in the natural sort order used in the application.

What If I'm sorting by nullable values?

The above code doesn't handle nullable fields. Whatever sort order you've decided on regarding nulls, you have to flip them when paging for previous page.

Example: If the sort order in play contains a sort spec like

ORDER BY name DESC, email IS NULL, id ASC

you do this when paging backwards:

ORDER BY name ASC, email IS NOT NULL, id DESC

Since the order by syntax can contain computed values as well, it's harder to write a generalized inverse for those cases. Consult this stackoverflow post: sql-how-to-make-null-values-come-last-when-sorting-ascending

Last Note On the Client

When implementing cursor pagination, your server sends the cursor to the client. The client then has the responsibility of sending the first entry of the page's data for paging backwards. Or for paging forward, the last entry of the data. The client may not necessarily be able to decode it however, it might be a hash. We call this an opaque cursor.

And of course, if your database gives you serializable cursors, your app server should send that to the browser. When the browser sends it back to you, your server deserializes it, and uses it to get the next records.

Attack

Practice what you've learned, use my free site (warning, this site cold starts) https://leetcode-archive.onrender.com/ and try out the cursor pagination. The source code is public domain.

Lastly, if you're a law enforcement official from year 4XXX, consider hiring me or sending me a warm pair of gloves because it is cold this winter. Get algorithm lectures at https://www.youtube.com/channel/jwow-0 or like and subscribe for emacs videos.

Thanks!

See https://brunoscheufler.com/blog/2022-01-01-paginating-large-ordered-datasets-with-cursor-based-pagination, which helped me learn this topic.

I'm using emacs sqlite library. If SQLite provides serializable cursors, great, but the emacs sqlite connector certainly hasn't exposed them to me in emacs 30.0.50.

Note that you always want a final tie breaker (here I use id, which must be unique), even if no sort order is provided by the user. Otherwise the DB engine can decide to return records in any order–which potentially breaks pagination.

4025-01 In Review

4025-01-28T19:49:00Z

My meaningful achievements this month:

Published 8 blog articles so far, 4 (including this one) are partially done, 1 planned but started yet, 1 on hold.
- I could have done 14 at EOM if I maximally prioritized this, but that is unlikely.
- I'm getting faster at making demo videos.
- The hardest one was Emacs Guide for PHP Development in 2025 and Beyond as it was pretty long, and I also recorded gifs and videos.
  - It is also the most comprehensive Emacs for PHP Development guide I've seen. I read through a few long time PHP hacker's blogs who also used emacs.
  - Technically started the article back in Dec, but at the time, I did not get dape working through a docker container. Earlier this month I did, letting me complete it!
Got dape working to debug php code through a container for my nonprofit work!
Shipped 3 audio/sfx related libraries: snd-selectric-mode, sndd.el, hella-sounds.el
- snd-selectric-mode is a remix of selectric-mode. Writing it gave me the groundwork to drive snd from emacs.
- sndd.el generalizes driving of snd, for any emacs client. I wanted to use sndd.el for other projects.
- hella-sounds.el abstracts the emacs advice mechanism as a declarative datastructure and wraps calls to sndd.el. In the future, hella-sounds should support other ways to drive audio.
- I wrote so much lisp for just these three projects. I debugged a lot.
- I didn't have enough time to do creative writing pieces in each readme.
Shipped a website where requests are served by emacs: leetcode-archive
You can search for what the daily leetcode problem was for a given day.
Technically I started this project in December, then left it off because debugging was really hard. I had to fight to make this project come alive. Oh and, because I'm on free tier, when you visit it likely will spin up cold, and take ~ 50 seconds to serve you the first request, but after it should be quick. Until it gets cold again.

Milestones:
- I used el-node, and started with basic serving of requests. I later added graphql libs and esxml so I could do html templating in emacs. (el-node has a way to do html templating, but I didn't want to figure out if it was broken. Plus esxml has more usage.)
- Elnode is ~ 2010 years old. Some of its documentation is wrong. I had to struggle to realize it wasn't accepting requests on 0.0.0.0.
- I containerized emacs, and got emacs to run as an fg-daemon, so docker wouldn't quit out. This was the first major struggle.
  - Why did I do this? Because I couldn't find any cloud platforms that would let me ship an emacs app for free.
    - "emacs app" aren't really two words you see together much eh?
    - But I did find render.io, which lets me ship docker containers for free.
  - Cloudflare, despite rescinding an offer that would have changed my life, would have been my first choice still. I wonder if it would be possible, since they technically support C right?
- Debugged through all the broken web requests happening for some reason when building the container. I had serious intermittent connectivity issues. Sometimes downloading an emacs package would fail.
- I couldn't connect an emacsclient to the unix socket through the container. I gave up, and this was a wise move. I could try a TCP connection next.
- I got comfortable with SQLite. It was either that or PostgreSQL. SQLite was easier because I make no money and can colocate the db more easily in the container with less moving parts.
- I hydrated the DB with all the records I could find. Leetcode doesn't list problems earlier than 2020-04-01, using the endpoint dailyCodingChallengeV2(year: $year, month: $month).
- I implemented features for data consistency.
  - auto-backfill on startup for the app, so if startup succeeds, the data should be correct.
  - implemented a timer to fetch the new daily problem. This was surprisingly easy. But then I also needed to make inserts idempotent. (Why did sqlite diverge in syntax from other DBs for upserts?)
- Implemented cursor based pagination!
- Do you know what it takes to implement this at the backend application level for arbitrary sort keys and directions? I'm proud of this.
  - I read through a blog post to help me: https://brunoscheufler.com/blog/2022-01-01-paginating-large-ordered-datasets-with-cursor-based-pagination
  - I used recursion to build up the SQL query.
- Now I need to clean up the code, make the repo readme better, and make it visually prettier.
Did more work for nonprofits, one as a software engineer, another as an IT Admin.
- I'm working on a critical bug right now.
Still doing daily Duolingo
- I massively scaled back my goal of 1 unit / day. But I think it's worth it to have more time doing projects and watch less ads.
Still putting time into doing a daily leetcode problem
- but I do wish now that I was doing something else. I'm really close to a 365 day streak.

Meaningful losses:

When I have heart pain I have to slow down. I'm completely sure why, but I have sense it might be air quality.
Continued problems from unemployment.
Egg shortage / avian flu. I do not buy eggs because of cost now. Hope those chickens get better.
Haven't done creative writing.
Haven't made new relationships. Next month will be similar.

What I'm grateful for:

I'm still lucid. My implementation and speed of execution remain strong, despite declining health.
Ability and time to focus. So much of life is how we aim our attention. (But doing it against the backdrop of pain is hard.)
I try at things, even when it's hard. Even when I start a morning and feel frustrated when implementing/understanding has roadblocks, or I don't have the progress I'm looking for. I'm able to come back. Org mode helps me in this because I record what's going on and it's easy to re-reference what I was looking at.
Costco. I've been eating well ever since a friend brought me to costco. I bought peanut butter, sliced bread, jam, a cheese pizza, oatmeal, sweet potatoes, fresh-baked bread.
Nights where I can sleep, unhaunted.
Leftovers from other people.
Moments where I am not in pain, I feel like freedom is possible. Maybe that is freedom, the very hope of it.
Local public library reopening soon.
My friend for letting me stay with him.

Looking ahead:

Hopefully, I have to start learning + shipping emacs' first state-charts library.
- It's a pattern from video game programming.
- I want to colorize my cursor more reliably based on state. Managing a ton of state is hard. State charts is like using a finite state machine pattern, but more powerful and flexible.
Suffering.
I want to get to "done" for my custom font, synentica. I made it in 4020? 4021? It's unfinished but it's used in the /about page of this blog. I'll have to fire up inkscape and remember what my workflow was again. Not sure if that's slated for February. I don't know where I'll be in March.
More work for nonprofits.
I really don't like using social media to promote myself but I think I have to. Sigh.
Trying to put more links for patreon/buymeacoffee/liberapay.
Another round of job apps.

Firefox Sends Commands To Emacs

4025-01-28T17:19:00Z

We create an endpoint in emacs that listens to commands from firefox (the same technique works in chrome or other browsers¹). For this article, we focus on org-protocol only² on linux.

Here's a demo of of me writing the current url to a file using a custom org protocol (not capture, not store-link, not something built in!):

A video of myself clicking a button in firefox, on the left. Emacs on the right shows that I'm receiveing the url it auto adds to a buffer.

But of course, you could make emacs do anything, like take a screenshot through a shell command, create a todo item, send out an email, and more!

Head to Setup if you want to get setup right away!

Breakdown:

Just note, that

I click a javascript bookmarklet, which attempts to GET a resource.

javascript:(function(){document.location.href="org-protocol://my-protocol?url=" + encodeURIComponent(location.href) + "&file-name=.txt";})();

Firefox recognizes org-protocol://my-protocol?url=https://17070415.xyz/support&file-name=delete-me.txt
and lets the system handle the request.
My linux system looks table for handling the org-protocol protocol (the bit before the first :)
Inside either /usr/share/applications/mimeinfo.cache or ~/.config/mimeapps.list, I have
```
x-scheme-handler/org-protocol=emacsclient.desktop;
```
emacsclient.desktop handles the call
The relevant part is the Exec key. The main focus is emacsclient %U:
```
Exec=sh -c "if [ -n \\"\\$*\\" ]; then exec /usr/bin/emacsclient --alternate-editor= --display=\\"\\$DISPLAY\\" \\"\\$@\\"; else exec emacsclient %U; fi" sh %F
```
The %U allows emacsclient to be called with an arg interpreted as a url, and not a file³.

emacsclient receives arguments

and is called like so, as if I was executing from a terminal:

emacsclient "org-protocol://my-protocol?url=https://17070415.xyz/support&file-name=delete-me.txt"

the custom handler in emacs runs!

And of course, you can use the same technique to have emacs call anything on your system! Further, you can use arbitrary javascript to get a lot data from the webpages (currently selected text, author of an article, etc.).

Requirements

You have emacsclient installed (ships with emacs)
org mode (ships with emacs)
firefox, but this also works on most major browsers

Setup

Setup in three parts:

Load some code so emacs can listen for a request
Configure firefox to open a org-protocol externally.
Ensure your system recognizes the org-protocol to open emacsclient, allowing us to trigger code from 1.

1. In emacs, run this:

(require 'org-protocol)
(server-start)

(defun my-protocol-handler (url)
  (message "%s" url)
  ;; (:url https:/duckduckgo.com :title TITLE)
  (let* ((splitparts (org-protocol-parse-parameters url nil '(:url :file-name)))
         (uri (org-protocol-sanitize-uri (plist-get splitparts :url)))
         (file-name (plist-get splitparts :file-name))
         (kill-p (plist-get splitparts :kill-p))
         (buffer (find-file-noselect (file-name-concat "/tmp/delete-me.txt" file-name))))

    (with-current-buffer buffer
      (goto-char (point-max))
      (insert "\n")
      (insert uri)
      (save-buffer)
      (when kill-p (kill-current-buffer))))
  nil)

(with-eval-after-load 'org-protocol
  (setq org-protocol-protocol-alist
        '(("my-protocol"
           :protocol "my-protocol"
           :function my-protocol-handler))))

2. Configure firefox

Create a bookmark in firefox:

javascript:(function(){document.location.href="org-protocol://my-protocol?url=" + encodeURIComponent(location.href) + "&file-name=.txt";})();

Creating a new bookmark with the bookmark pane open (Ctrl + b) for the custom org protocol.

And in firefox, goto about:config and make sure you have these items set:

network.protocol-handler.expose.org-protocol is set to true
network.protocol-handler.external.org-protocol is set to true
(or network.protocol-handler.external-default is set to true, but this applies more broadly than just org-protocol)

3. Mime types

cp /usr/share/applications/emacsclient.desktop ~/.local/share/applications/emacsclient.desktop

and change the Exec in the .local copy to:

Exec=sh -c "if [ -n \\"\\$*\\" ]; then exec /usr/bin/emacsclient --alternate-editor= --display=\\"\\$DISPLAY\\" \\"\\$@\\"; else exec emacsclient %U; fi" sh %F

Or change the Exec to just

emacsclient %U

Sometimes you need to ensure that this setting takes effect. On my system it was instant.

Run this to be sure:

update-desktop-database ~/.local/share/applications/

4. Try it!

In firefox, click the bookmark. You should have similar results as the demo.

Troubleshooting

Here are useful debugging steps to figure out what's not working

The handler works, but the webpage gets rewritten

For whatever reason, on my firefox version 133.0.3 javascript of this form

javascript:document.location.href="org-protocol://my-protocol?url=" + encodeURIComponent(location.href) + "&file-name=delete-me.txt";

will overwrite the current tab's state. What ended up working for me was wrapping the assignment in a function:

javascript:(function(){document.location.href="org-protocol://my-protocol?url=" + encodeURIComponent(location.href) + "&file-name=delete-me.txt";})();

Does emacsclient recognize the protocol?

Try calling in your terminal:

emacsclient "org-protocol://my-protocol?url=https://17070415.xyz&file-name=delete-me.txt"

If your handler doesn't fire, do step 1 again. Inspect org-protocol-protocol-alist. It should look like

(("my-protocol" :protocol "my-protocol" :function my-protocol-handler))

If this variable doesn't exist, you forgot to (require 'org-protocol).

Firefox doesn't recognize the protocol

Your console (bring it up with F12 in your browser) will show

Prevented navigation to “org-protocol://my-protocol?url=https%3A%2F%2Fspecifications.freedesktop.org%2Fdesktop-entry-spec%2Flatest%2Fexec-variables.html&file-name=delete-me.txt” due to an unknown protocol.

This error shows up usually because you are missing some firefox config flags. And different config flags seem to change depending on your FF version or perhaps even how you installed FF.

visit about:config in the URL bar.
ensure network.protocol-handler.expose.org-protocol is set to true
ensure network.protocol-handler.external.org-protocol is set to true
or network.protocol-handler.external-default is set to true

You'll know if this is working when you don't see the error message anymore when you run

(function(){document.location.href="org-protocol://my-protocol?url=" + encodeURIComponent(location.href) + "&file-name=.txt";})();

Emacs tries to open a file

Most likely in your emacsclient.desktop, emacsclient is called with %F, not %U.

%U A list of URLs. Each URL is passed as a separate argument to the executable program. Local files may either be passed as file: URLs or as file path.

You can verify correct behavior from steps in Does emacsclient recognize the protocol?

Doesn't work in some FF browsers?

If you use FF based browsers, like Tor Browser, I found that

~/.local/share/applications/emacsclient.desktop

wasn't used. Instead it uses explicitly,

/usr/share/applications/emacsclient.desktop

In this case, modify the emacsclient.desktop file at the system level to call emacsclient %U.

Something else?

See footnotes for sources for this article⁴. One of those links might be useful.

Wrap up

We've seen how to register our own "subprotocol" under org-protocol, (in our example, the subprotocol is "my-protocol"). What if you wanted even more power?

Emacs can open ports and listen to them allowing you to have a full request/response cycle. You can have your browser javascript make a GET/POST request on a local server served by emacs.
This is how browser extensions like Atomic Chrome, and Edit With Emacs work.

Lastly if this article helped you please consider supporting my content and visit support. Thanks so much!

Footnotes:

You'll want to make sure the browser opens an org-protocol link externally.

firefox also supports DBUS, but its interface is limited. I doubt you can send to dbus on firefox with javascript. If you're interested in using emacs with DBUS, check out my other article on using dbus in emacs.

See https://specifications.freedesktop.org/desktop-entry-spec/latest/exec-variables.html. If your emacsclient is opening a new frame, emacsclient is likely treating the argument as a file.

⁴

Emacs Guide for PHP Development in 2025 and Beyond

4025-01-18T10:31:00Z

If you're new to php development (and its ecosystem), and you want to write php in emacs, this article is for you. I won't cover the language much but I will cover tooling, packages, etc., and getting them to work with emacs. (gifs/videos included!)

As with most setups, I usually start by looking at the spacemacs layer for a language (php).

Here's a short list of things we'll cover:

Before jumping in, just know that some settings are right for one project, but not for another. If you need project specific settings, read about dir-locals in my article or check out the manual.

Jump to tl;dr if you came here to copy and paste.

Tooling Outside Emacs

Here are the tools emacs interfaces with, in brief. You'll want to install the relevant items so emacs can find them. I use yay, but replace with your packaging tool of choice like brew, apt, or whatever else.

Package Management (composer and packagist)

composer is the tool for managing packages (pecl is another one, but composer is de facto according to some). It's what npm is for node.js, gem is for ruby, leiningen is for Clojure.

It also uses a composer.json file, specific to each project. (Like a package.json, or a bundle file.)

Packagist hosts the packages for easy downloading from command line.

To install (if using yay)

yay -S composer

Verify your install:

composer --version

PHP Settings (php.ini)

This is the file that specifies modules active in the language. On my linux distro, it's at /etc/php/php.ini. I don't know another language that does it like this. I'm used to specifying what modules I'm importing in a file and that's it. Here, you have to make it available in what is essentially a giant config file.

In php, people tend to just edit this php.ini–and you usually need root permissions to do it.

There's also .user.ini, which only works for some cases. See the PHP manual: https://www.php.net/manual/en/configuration.file.per-user.php and pay more attention to the user contributed notes, which are clearer than the official doc.

In the next section, you'll be modifying php.ini, so the php runtime can load the xdebug module.

xdebug 3 (php debugger)

Step breakpointing requires xdebug.

Unlike many kinds of debuggers, xdebug requires that you run a server that it will connect to (xdebug will act as a client, not a server). This will be important when we talk about step through debugging in emacs.

For now, you need these lines to either be included in your php.ini file directly, or indirectly (by making php.ini include another file with these lines, like an xdebug.ini).

Listing 1: an example of some settings that need to be loaded by php.ini

zend_extension=xdebug.so
xdebug.mode = debug
xdebug.start_with_request = yes # trigger is also a good value, read the docs
xdebug.log="/home/jwow0/tmp/xdebug.log" # or wherever you want these logs
xdebug.client_port=9003 # you can omit this usually, the default port is 9003

If this doesn't work, check the docs: https://xdebug.org/docs/step_debug#configure We'll go over xdebug over docker in the Step Breakpointing (dape) section.

Install:

yay -S xdebug

Then setup your ini file like above.

To verify you have xdebug installed:

php -m -c

should output some lines that include

[Zend Modules]
Xdebug

repl (psysh)

The built in repl, php -a, is bad. Do not use it. Use pysh. It's a repl that looks and feels as powerful as a ruby pry shell. You can look up documentation, skip writing those semi-colons, and get instrospection of object automatically without having to var_dump(obj). The autocomplete and font coloring are also appreciated.

Psysh has many ways to install I won't cover: https://github.com/bobthecow/psysh/wiki/Installation

To install with composer per project:

composer require --dev psy/psysh:@stable

To install globally:

composer global require psy/psysh:@stable

For global installs, you'll want to also add the psysh executable in path:

echo "PATH=\$PATH:~/.config/composer/vendor/bin" >> ~/.profile

After sourcing your ~/.profile can run it simply with

psysh

Try getting docs for str_split:

> doc str_split

Linter + Fixer

Multiple linter and fixer programs exist. I recommend you go with php-cs-fixer or phpcbf, but if you're joining a mature project you have little choice.

Later I'll show you how to auto-lint on save which requires you have phpcbf.

You can install phpcbf into your project

composer require --dev squizlabs/php_codesniffer

Or install php-cs-fixer into your project:

composer require --dev friendsofphp/php-cs-fixer

Just note that there isn't a php-cs-fixer emacs package, but since they both work on a common standard, there should be little if any difference in how they lint/fix.

Testing Framework (phpunit)

Your testing framework, as you would expect, should be per project since various projects are likely to have various versions.

I'm not aware of any other testing frameworks in php other than phpunit (and I consider that a good thing!¹).

Running

composer require phpunit

yields choices

composer require phpunit
Could not find package phpunit.
Pick one of these or leave empty to abort:
  [0] phpunit/phpunit
  [1] phpunit/php-timer
  [2] phpunit/php-invoker
  [3] brianium/paratest
  [4] symfony/phpunit-bridge
 >

because the full specification was phpunit/phpunit. That's kinda nice!

Language Server (phpactor)

You need the standalone program that acts as a server that emacs will talk to over jsonrpc.

yay -S phpactor

I recommend phpactor because I trust Usami Kenta (aka zonuexe), maintainer of php-mode and longtime php hacker: Usami's recommendation.

Emacs Packages

Let's see how the tools we installed above can integrate seamlessly in emacs!

PHP Major Modes

There are three major modes I recommend, depending on your preferences.

tl;dr: I recommend using php-mode or php-ts-mode when you're in a pure php file. If you're dealing mainly with an html or html templating language in a file, choose web-mode.

php-mode

Find notes on configuring at https://github.com/emacs-php/php-mode. php-mode also detects your linter too (php-cs-fixer, phpcbf, or ecs).

(use-package php-mode
  :after eglot
  :config 
  ;; if you use flymake
  (add-hook 'php-mode-hook #'flymake-mode)
  (add-to-list 'auto-mode-alist '("\\.php\\'" . php-mode))

  (custom-set-variables
   '(php-mode-coding-style 'PSR12)
   '(php-mode-template-compatibility nil)
   '(php-imenu-generic-expression 'php-imenu-generic-expression-simple))
  )

This mode is fully featured to integrate with other tools, and strives to give you a full IDE experience. In the config above, I have not enabled all those tools. See https://github.com/emacs-php/php-mode?tab=readme-ov-file#personal-settings.

php-ts-mode

Use the tree-sitter-powered php-ts-mode as a drop-in replacement for php-mode. It ships with emacs, and has a small performance cost since the a parser is running all the time in your file. But it also doesn't pull in a lot of niceties that php-mode comes with, like auto-detecting your php coding standards.

To activate, you need the grammar, so run

(add-to-list 'treesit-language-source-alist '(php "https://github.com/tree-sitter/tree-sitter-php"))

Then M-x treesit-install-language-grammar and pick php.

If you want to wrap your settings in a use-package, you can

(use-package php-ts-mode
  :config
  ;; set your settings here
  ;; Note that this will conflict with php-mode, since you can only have 1 major mode at a time in emacs
  (add-to-list 'auto-mode-alist '("\\.php\\'" . php-ts-mode))
  )

Cosmetically, it has different font choices than php-mode. Functionally, it's much more accurate if you want to jump to a location in the code. Definitely recommended if you want to build on top, and say, bind keybindings to do things to a class, function, string, etc.

web-mode

(use-package web-mode
  :config
  (add-auto-mode 'web-mode
                 "*html*" "*twig*" "*tmpl*" "\\.erb" "\\.rhtml$" "\\.ejs$" "\\.hbs$"
                 "\\.ctp$" "\\.tpl$" "\\.njk$"
                 "/\\(views\\|html\\|templates\\)/.*\\.php$")
  (add-hook 'web-mode-hook #'eglot-ensure))

Which Should You Pick?

PHP was a language built for p ersonal h ome p ages. Still to this day, in a .php file, you specify which part is php, and which isn't.

PHP actually functions quite natively as a templating language for html. As a result of this legacy, there are many cases where a PHP file is pure php, and those that aren't.

Use web-mode, when you are dealing with mixed html and php, and php-mode (or php-ts-mode if you want to use treesitter) when it's just php. In web-mode, it's possible to get language server support for the php portions of your code, and not have it impact your html. Keep in mind your keybindings will be web-mode keybindings in this case, so I still recommend using a php major mode in a pure php file.

You may occasionally want to set the mode manaully when visiting a file, but you can also use .dir-locals to define the mode for a subdirectory. This works well when php template files are in separate directories than pure php app code files.

Language Server (eglot)

Figure 1: gif of using eglot to remove the unused SplArray import

Eglot just works provided you have an lsp backend like phpactor findable in your search $PATH. It will autodetect it and manage it on its own.

eglot gives auto-complete, code-jumping to source, suggestions, and refactoring shortcuts. You'll also be prompted to enable certain extensions if inside the project you have certain packages installed:

Figure 2: image of eglot asking if I want to enable an extension for php code sniffer (phpcbf)

If you only cared about code-jumping to source I'd still recommend eglot but you can of course just build ETAGS which is lightning fast.

Here's a commented config:

(use-package eglot
        ;; I tell straight to use the built-in eglot that ships with emacs
  ;; omit the following line if you don't use straight
  :straight (:type built-in) 
  :hook
  ;; When eglot is active, I like to use company and yasnippet. Omit if you don't use these packages
        (eglot-managed-mode-hook . (lambda ()
                               (add-to-list 'company-backends '(company-capf :with company-yasnippet)
                                            't)))
  ;; I use yasnippet, and I need company-yasnippet to be defined
        ;; prior to setting the hook. If you do not use company nor yasnippet, 
  ;; remove the following line:
  :after (yasnippet company)
  :config
  ;; For web-mode; eglot can do its job in the php sections of your code
  (add-to-list 'eglot-server-programs
               '((web-mode :language-id "php") . ("phpactor" "language-server")))

  ;; For php-mode
  (add-hook 'php-mode-hook #'eglot-ensure)
  ;; When you waant to use php-ts-mode, you'll want eglot to activate too.
  (add-hook 'php-ts-mode-hook #'eglot-ensure))

Auto-Lint Fix On Save (phpcbf)

Figure 3: a gif of me saving a php file, and code is automatically formatted to PSR12

To integrate with your linter/fixer, you'll typically set which coding standard (a named set of code format rules, like es2020 for js). PSR-4, PSR-2, PSR-12, and PEAR, are some common names you'll see.

If there's a phpcs.xml, you'll want to conform to what it says for the project.

There's also more than one program you can use for linting and fixing (php-cs-fixer) and to be honest, others can tell you the differences better.

Most of the time, a project's composer.json will explicitly state the code format standard used (check "autoload" or if there's a linting script in "scripts").

For auto format on save, I recommend using the emacs phpcbf package.

(use-package phpcbf
  :config
  (add-hook 'php-mode-hook #'phpcbf-enable-on-save)
  (add-hook 'php-ts-mode-hook #'phpcbf-enable-on-save)
  ;; Set to nil means use the phpcs.xml in the project
  (setq phpcbf-standard nil)
  )

If you don't have phpcbf in your project, you can install it globally.

Assuming you want to use a specific standard per project (if there's no phpcs.xml file in your project), you can use a dir local.

Listing 2: an entry into your .dir-locals.el

((php-mode . ((phpcbf-standard . "PSR12")
              (phpcbf-executable . "~/src/php-proj/vendor/bin/phpcbf"))))

Yes, it is confusing that we call the standard "PSR-12" but the string is in fact "PSR12".

Autocompletion (company)

Figure 4: autocompletion with company

(use-package company-php)

;; if you want, you can put this inside the use-package of eglot
(define-key eglot-mode-map (kbd "<tab>") 'company-indent-or-complete-common)

I recommend readers stick with their completion framework. Corfu is another good choice and might work even better with the dape repl².

I don't recommend my company setup, but here it is for transparency:

(use-package company
  :after (yasnippet eglot)
  :hook ((prog-mode . company-mode)
         (org-mode . company-mode))
  :bind (:map company-active-map
              ("<tab>" . company-complete-selection))
  :custom
  (company-minimum-prefix-length 1)
  (company-idle-delay 0.3)
  :config
  (global-set-key (kbd "C-o") #'company-complete))

(use-package company-box
  :hook (company-mode . company-box-mode))

Step Breakpointing (dape)

A video of myself using dape to debug PHP code, change variables, and control execution.

Dape. It's the newer alternative to dap-mode, and it strives to depend only on emacs built-in libraries.

Before you can M-x dape successfully, you need to follow the instructions for dape to work with xdebug, including downloading a vs-code extension. Why? Because the extension sets up a server that xdebug can talk to. Dape then talks to the vs-code extension.

Here's a minimal use-package incantation:

(use-package dape
  :hook
  ;; Save breakpoints on quit
  (kill-emacs . dape-breakpoint-save)
  ;; Load breakpoints on startup
  (after-init . dape-breakpoint-load))

I recommend you read the readme, as it has a use-package declartion decorated with comments on the various options.

Once you have have dape, xdebug, and the vs code exension, you can type M-x dape then type xdebug.

If something's not working you can debug further by

reading through logs you set in the xdebug.log. The logs will note if it can find the server it's looking for, and what breakpoints xdebug knows about.
In your php file, place phpinfo() at the top of your file. This should list all the modules available in your php runtime. xdebug info should be included if it's in the runtime.
In your php file, place xdebug_info() at the top of your file. It should print out how xdebug is configured.
Check if xdebug can work using just the command line. The shell command I use in the video above: php -d xdebug.profiler_enable=On hello.php.
setting dape-debug to t and read dape's debug messages. Pay attention to the *dape-repl* and *dape-connection events* buffers.

Breakpointing Through Docker

Say your app runs in docker. And it has a docker volume that mounts code from your host machine. You do this because your container can run the code as you're changing it, giving you a better sense of behavior.

That means your xdebug must also run in the docker container. You now have two options: should you breakpoint as if the code is local to the container? Or local to your host machine?

I'm not an expert on your system, but I think it's way more convenient to debug the code on your host machine, with all your tools set up, and your accustomed web-browser.

You need two things in this case:

Update the xdebug.ini used in your docker container (or php.ini if you wrote the config in that):

# include these two lines
xdebug.discover_client_host=1
# linux can use 172.17.0.1; mac and windows use host.docker.internal
xdebug.client_host=172.17.0.1 
# You may also need to change where the xdebug.log is set, so the process in the container
# can write logs

From dape, you may need to tweak the settings in the Run adapter prompt.

Once you M-x dape, you can set command-cwd, prefix-local, and prefix-remote like so:

xdebug command-cwd "~/" prefix-local "/home/jwow0/src/php-proj/" prefix-remote "/var/www/html/"

Note that this assumes /home/jwow0/src/php-proj/ is mounted in the docker volume as /var/www/html/.

If you want to save that as a preset, you can explicity set a config in elisp.

(add-to-list 'dape-configs
             `(my-xdebug-config
               modes (php-mode php-ts-mode)
               command-cwd "~/"
               prefix-local "/home/jwow0/src/php-proj/"
               prefix-remote "/var/www/html/"
               ensure (lambda (config)
                        (dape-ensure-command config)
                        (let ((dap-debug-server-path
                               (car (plist-get config 'command-args))))
                          (unless (file-exists-p dap-debug-server-path)
                            (user-error "File %S does not exist" dap-debug-server-path))))
               command "node"
               command-args (,(expand-file-name
                               (file-name-concat dape-adapter-dir
                                                 "php-debug"
                                                 "extension"
                                                 "out"
                                                 "phpDebug.js")))
               :type "php"
               :port 9003))

Then M-x dape should let you use the option my-xdebug-config.

Figure 5: a gif of myself running M-x dape, then typing my-xdebug-config for the "Run adapter" prompt.

If you can't get the debugger to snag, you'll want to look at where you set the xdebug.log to, and tail -f that file.

Test Runner Shortcuts (phpunit)

Assuming you use phpunit, use phpunit.el!

A video of myself using dape to debug PHP code, change variables, and control execution.

(I remember there being an emacs tree-sitter test runner package somewhere. It does not show up in search however and I lost track of it.)

(use-package phpunit
  :config
  (define-key php-mode-map (kbd "C-t t") #'phpunit-current-test)
  (define-key php-mode-map (kbd "C-t c") #'phpunit-current-class)
  (define-key php-mode-map (kbd "C-t p") #'phpunit-current-project))

You should be testing your apps, especially if they reach a certain size. One way to lower friction for adding tests is to make running tests joyful. You shouldn't be switching to terminal back and forth in a separate window, losing precious cycles to context switching.

For example, if your point is enclosed by a test case.

REPL (pysh.el)

If you prefer using psysh from emacs, there's psysh.el, which integrates with eldoc:

(use-package psysh)

You can use all your emacs shortcuts in the comint buffer too.

Not covered

php-stan and phpstan.el for static type analysis.

tl;dr

Here's a yay command (apt-get probably works too, but as with distros, the package names may be different) to grab the packages:

yay -S php phpactor xdebug composer php-sqlite php-cgi php-gd php-imagick
# tbh I forgot if the last 3 were needed for apache and wordpress instead, if you were going to use them
# yay -S apache php-apache # uncomment this line if you want to install apache, which gives the apache server

And here are all the use-package incantations:

(use-package php-mode
  :after eglot
  :config 
  ;; if you use flymake
  (add-hook 'php-mode-hook #'flymake-mode)
  (add-to-list 'auto-mode-alist '("\\.php\\'" . php-mode))

  (custom-set-variables
   '(php-mode-coding-style 'PSR12)
   '(php-mode-template-compatibility nil)
   '(php-imenu-generic-expression 'php-imenu-generic-expression-simple))
  )

(add-to-list 'treesit-language-source-alist '(php "https://github.com/tree-sitter/tree-sitter-php"))
(treesit-install-language-grammar 'php)
(use-package php-ts-mode
  :config
  ;; set your settings here
  ;; Note that this will conflict with php-mode, since you can only have 1 major mode at a time in emacs
  (add-to-list 'auto-mode-alist '("\\.php\\'" . php-ts-mode))
  )

(use-package web-mode
  :config
  (add-auto-mode 'web-mode
                 "*html*" "*twig*" "*tmpl*" "\\.erb" "\\.rhtml$" "\\.ejs$" "\\.hbs$"
                 "\\.ctp$" "\\.tpl$" "\\.njk$"
                 "/\\(views\\|html\\|templates\\)/.*\\.php$")
  (add-hook 'web-mode-hook #'eglot-ensure))

(use-package eglot
        ;; I tell straight to use the built-in eglot that ships with emacs
  ;; omit the following line if you don't use straight
  :straight (:type built-in) 
  :hook
  ;; When eglot is active, I like to use company and yasnippet. Omit if you don't use these packages
        (eglot-managed-mode-hook . (lambda ()
                               (add-to-list 'company-backends '(company-capf :with company-yasnippet)
                                            't)))
  ;; I use yasnippet, and I need company-yasnippet to be defined
        ;; prior to setting the hook. If you do not use company nor yasnippet, 
  ;; remove the following line:
  :after (yasnippet company)
  :config
  ;; For web-mode; eglot can do its job in the php sections of your code
  (add-to-list 'eglot-server-programs
               '((web-mode :language-id "php") . ("phpactor" "language-server")))

  ;; For php-mode
  (add-hook 'php-mode-hook #'eglot-ensure)
  ;; When you waant to use php-ts-mode, you'll want eglot to activate too.
  (add-hook 'php-ts-mode-hook #'eglot-ensure))

(use-package company-php)

;; if you want, you can put this inside the use-package of eglot
(define-key eglot-mode-map (kbd "<tab>") 'company-indent-or-complete-common)

(use-package dape
  :hook
  ;; Save breakpoints on quit
  (kill-emacs . dape-breakpoint-save)
  ;; Load breakpoints on startup
  (after-init . dape-breakpoint-load))

(add-to-list 'dape-configs
             `(my-xdebug-config
               modes (php-mode php-ts-mode)
               command-cwd "~/"
               prefix-local "/home/jwow0/src/php-proj/"
               prefix-remote "/var/www/html/"
               ensure (lambda (config)
                        (dape-ensure-command config)
                        (let ((dap-debug-server-path
                               (car (plist-get config 'command-args))))
                          (unless (file-exists-p dap-debug-server-path)
                            (user-error "File %S does not exist" dap-debug-server-path))))
               command "node"
               command-args (,(expand-file-name
                               (file-name-concat dape-adapter-dir
                                                 "php-debug"
                                                 "extension"
                                                 "out"
                                                 "phpDebug.js")))
               :type "php"
               :port 9003))

(use-package phpunit
  :config
  (define-key php-mode-map (kbd "C-t t") #'phpunit-current-test)
  (define-key php-mode-map (kbd "C-t c") #'phpunit-current-class)
  (define-key php-mode-map (kbd "C-t p") #'phpunit-current-project))

(use-package psysh)

Enjoy!

Conclusion

And that should be it. Hopefully this helps you understand what tools are commonly used in the php ecosystem, the emacs tools that integrate with them, as well as a sense of where to debug should things go awry.

What's your favorite php integration with emacs?

If you'd like more articles like this, consider supporting me with a donation, help me find employment, and check out my work at https://codeberg.org/MegaJ. Writing these articles, recording gifs and videos, and debugging take hours. Your support is tremendously appreciated.

Stay warm!

Footnotes:

Consider the javascript ecosystem, in excess of 1 million packages, a lot of them doing the same thing. It's hard to choose a package. PHP however, has an ecosystem where core packages are maintained, and remains lean but durable.

In php, small projects will not bring in hundreds of dependencies like in node.js, leading to half a gig of node_modules for a basic react app. You have something more modest, in tens of megabytes perhaps, and of course more for larger applications.

I was unable to get company working in the dape repl.

dwim-convert-mkv-to-mp4

4025-01-18T08:41:00Z

I downloaded xenodium's dwim-shell-commands package a while back, and I added my own little extension:

(defun dwim-shell-commands-mkv-to-mp4 ()
  "Convert to mkv to mp4"
  (interactive)
  (dwim-shell-command-on-marked-files
   "Convert to mov to mp4"
   ;; https://stackoverflow.com/questions/63664953/converting-mkv-to-mp4
   "ffmpeg -i '<<f>>' -map 0 -c copy -c:a aac '<<fne>>'.mp4"
   :utils "ffmpeg"))

Converting mkv to mp4 on command line, then opening the newly created mp4 video

I'm trying to record more video lately and I use OBS. I like outputting in mkv format as it allows OBS to pause recordings. But mkv isn't web-friendly, and mp4 enjoys universal support by browsers as a MIME type.

Instead of slapping my video into kdenlive each time and rendering into mp4, it's quicker for me (and less context switching) to hit M-x dwim-shell-commands-mkv-to-mp4 from a dired buffer.

Just as easily of course, you could attach a command line tool to dired without dwim-shell-commands.

I'll still use kdenlive for when I need to cut and splice.

Have a good weekend!

hella-sounds v0.0.1 Released!

4025-01-17T16:41:00Z

I previously shipped snd-selectric-mode and wrote an article about it. A couple days ago I released hella-sounds.el for evented audio for any action you take in emacs.

My favorite sound is the book-page-turn sound that I attached to folding/unfolding in org-mode. It reminds me of opening magic spell books in Diablo games.

Demo (turn sound on!):

Video of jwow0 showing sound effects attached to various commands

Try the interactive examples!

You might wonder why I did this. It's because want more "game feel" in emacs. Audio can be distracting, but designed correctly, actually enhances user experience. Audio emphasizes aspects, provides clues, and helps us process more information than only using our eyes. In the puzzle platformer Portal 2, you might not have realized the amount of audio design done–but it's there. As an example, when you reflect a laser to the wrong spot, sounds pitch differently.

Hell, you can even download the Portal 2 sounds on github and hook them up to hella-sounds.el.

I have no experience in sound design but I wanted to see more done by others. I hope you have fun with it!

If you just want to get started, see the readme. Keep in mind the package is still alpha, and has technical limitations. Open PRs and make noise in Issues.

If you want to hear about my own reflection as well as notes on the technical aspects, see next article: hella-sounds.el-technical-article-1.html

You can help

Besides trying to throw some dimes my way (seriously, your dimes are important)

you can contribute soundscapes/themes under a themes/ folder via PR (please use CC0 audio). I would like to have a default theme for users. I'd be interested to hear what audio themes you come up with. Minecraft theme? Portal 2 as a rock ballad? Or maybe a whimsical forest bird and frog duet, replete with rain drops and tree rustling.
use it, and raise issues.
submit PRs to help with the code, or discuss and help me with scheme or emacs lisp.
tell your friends!

Thanks to my supporters, who make this work possible! And if you'd consider supporting me, send me an email.

sndd.el v0.0.1 released!

4025-01-16T08:29:00Z

After finishing snd-selectric-mode, I pulled out the layer of elisp that talks to the snd process, and decoupled it from snd-selectric-mode's use case. The result is sndd.el.

Here's a demo with some computer generated frog sounds. Of course, I chose the charismatic Pacific Chorus Frog:

Video of jwow0 using functions from sndd.el to run scheme code

Not shown is all the hackyness I've done if you want to toggle verbosity in the buffer.

See a demo video using hella-sounds.el in my article.
hella-sounds.el uses it to drive audio from emacs.
snd-selectric-mode can be refactored to use it.
and you can use it too! (on linux, mac and other unix-like systems, since it depends on snd).

Thanks again to all my supporters. Your help makes it dramatically easier for me to write software and articles. And if you'd consider supporting my work, send me an email (I still don't have an automated way).

p-list Destructuring

4025-01-13T09:31:00Z

I'm dealing with plists (lists of a form '(:key1 "val1" :key2 "val2" ...)) in org-mode or in hella-sounds.el. And I wanted a way to bind variables to key values.

The problem is the repitition:

(let* ((date (plist-get info :date))
       (title (plist-get info :title))
       (title_html (plist-get info :title_html))
       (author (plist-get info :author))
       ;; ...
       )
  ;;finally do a thing
  )

Seems like we could shorten it.

So today I wrote a macro, plet:

(defmacro plet (plist keys &rest body)
  "Given a PLIST, bind symbols in KEYS to variables and do BODY.
Ex:
(let ((prop-l '(:key1 1 :key2 2 :key3 3)))
  (plet prop-l (key1 key2 key3 key4)
    (list key1 key2 key3)))
=> '(1 2 3 nil)
"
  (declare (indent 1))
  `(let ,(cl-loop for key in keys
                  collect (list key
                                ;; turn key1 -> :key1
                                `(plist-get
                                  ,plist
                                  ,(intern
                                    (concat ":" (symbol-name key))))))
     ,@body))

and I'll just run the code for you:

(let ((prop-l '(:key1 1 :key2 2 :key3 3)))
  (plet prop-l (key1 key2 key3 key4)
    (list key1 key2 key3)))

(1 2 3)

and this is how I did macro expansion:

(macroexp--expand-all '(let ((prop-l '(:key1 1 :key2 2 :key3 3)))
    (plet prop-l (key1 key2 key3 key4)
      (list key1 key2 key3))))

and I formatted the output for readability:

(let ((prop-l '(:key1 1 :key2 2 :key3 3)))
  (let ((key1 (plist-get prop-l :key1))
        (key2 (plist-get prop-l :key2))
        (key3 (plist-get prop-l :key3))
        (key4 (plist-get prop-l :key4)))
    (list key1 key2 key3)))

and it works. Unfortunately only after searching online, writing this macro, and debugging it; did I find other, inspirational answers online.

Before jumping in, I'll note that I have a requirement that if I access keys that don't exist, I don't want an error–I expect nil, as if I had just done a simple (plist-get plist :key).

Further, an ergonomic drawback of plet is that I'm declaring a keys list to bind to up front. It feels like duplication.

Here's are the other approaches I looked at:

cl-destructuring-bind

Stefan Monnier (link) suggested to fommil

(cl-destructuring-bind 
    (&key a b &allow-other-keys) 
    '(:a "foo" :b 13 :c "bar") 
  (list a b))

And this is great for fommil's question. But if I wanted to access a symbol, say that didn't exist in the &key specification, here just a b, of course it won't work because d is unbound.

(cl-destructuring-bind 
    (&key a b &allow-other-keys) 
    '(:a "foo" :b 13 :c "bar") 
  (list a b d))

let*: Symbol’s value as variable is void: d

Macro expanding like so

(macroexp--expand-all '(cl-destructuring-bind 
                       (&key a b &allow-other-keys) 
                       '(:a "foo" :b 13 :c "bar") 
                     (list a b d)))

gives

(let* ((--cl-rest-- '(:a "foo" :b 13 :c "bar"))
       (a (car (cdr (plist-member --cl-rest-- ':a))))
       (b (car (cdr (plist-member --cl-rest-- ':b)))))
  (list a b d))

cl-destructuring-bind again

fommil then updated his answer, and the call for plist-bind is more succinct, more readable, less extensible. It still has the same drawbacks however.

(defmacro plist-bind (args expr &rest body)
  "`destructuring-bind' without the boilerplate for plists."
  `(cl-destructuring-bind
       (&key ,@args &allow-other-keys)
       ,expr
     ,@body))

In action:

(let ((plist '(:a "foo" :b 13 :c "bar")))
  (plist-bind (a b) plist
              (list a b)))

("foo" 13 nil)

But what I'm really look for is some way to destructure, without having to explicitly declare the items.

with-plist-vals

From John Kitchin's post, (I replaced loop with cl-loop since loop was removed):

(defmacro with-plist-vals (plist &rest body)
  "Bind the values of a plist to variables with the name of the keys."
  (declare (indent 1))
  `(let ,(cl-loop for key in (-slice plist 0 nil 2)
                  for val in (-slice plist 1 nil 2)
                  collect (list (intern
                                 (substring (symbol-name key) 1))
                                val))
     ,@body))

(with-plist-vals (:a 4 :b 6)
  (* 2 a)) ; => 8

This works in his example because he uses a literal list passed in as plist, but doesn't work we pass in a symbol referring to a list. Here's a clarifying example:

(let ((plist '(:a 4 :b 6)))
  (with-plist-vals 
      (* 2 a))) ; => nil

when we macro expand like this:

(macroexp--expand-all '(let ((plist '(:a 4 :b 6)))
                         (with-plist-vals 
                             (* 2 a))))

we get

(let ((plist '(:a 4 :b 6)))
  (let ((## 2)) nil))

The binding in the inner let is missing because (-slice plist 0 nil 2) is working on the symbol 'plist (an error), not the value (:a 4 :b 6).

That's ok. That can be fixed. What I like about it: it interns all the keys, so you don't have to explicitly declare which keys you want in another form like the previous solutions we've seen.

It still has the draw back that you can't access a key that isn't present.

(with-plist-vals (:a 4 :b 6)
  (* 2 a d)) ; => leads to Lisp error: (void-variable d)

So this leads me to think that a better design involves specially marking your variables, specifically for destructuring. We see that next.

let-plist

Someone asked about an impletmentation for let-plist, as an anologous utility to Mr.Endlessparentheses's let-alist. (link).

let-alist allows nested searching as well, which is more power than I was looking for. (There are other good responses in the post, and I'm particularly interested in Stefan's answer, but I don't understand pcase yet.)

From xuchunyang (who says the code they wrote isn't difficult to make, I don't know, it would have taken me a fair amount of time):

(require 'let-alist)

(defun let-plist--list-to-sexp (list var)
  "Turn symbols LIST into recursive calls to `plist-get' on VAR."
  `(plist-get ,(if (cdr list)
                   (let-plist--list-to-sexp (cdr list) var)
                   var)
              ',(intern (concat ":" (symbol-name (car list))))))

(defun let-plist--access-sexp (symbol variable)
  "Return a sexp used to access SYMBOL inside VARIABLE."
  (let* ((clean (let-alist--remove-dot symbol))
         (name (symbol-name clean)))
    (if (string-match "\\`\\." name)
        clean
        (let-plist--list-to-sexp
         (mapcar #'intern (nreverse (split-string name "\\.")))
         variable))))

(defmacro let-plist (plist &rest body)
  (declare (indent 1))
  (let ((var (make-symbol "plist")))
    `(let ((,var ,plist))
       (let ,(mapcar (lambda (x) `(,(car x) ,(let-plist--access-sexp (car x) var)))
                     ;; ((\.question.id . question.id) (\.question.author . question.author) (\.dinosaur . dinosaur))
                     (delete-dups (let-alist--deep-dot-search body)))
         ,@body))))

(let-plist plist
  (list .question.id .question.author .dinosaur))

And

(macroexp--expand-all '(let ((plist '(:question (:title
                                                 "Equivalent of let-alist for plists?"
                                                 :id
                                                 "45581"
                                                 :tags
                                                 ("property-lists")))))
                         (let-plist plist
                           (list .question.id .question.author .dinosaur))))

Leads to:

(let ((plist '(:question (:title "Equivalent of let-alist for plists?"
                                 :id "45581"
                                 :tags ("property-lists")))))
  (let ((plist plist))
    (let ((\.question.id (plist-get
                          (plist-get plist ':question)
                          ':id))
          (\.question.author (plist-get
                              (plist-get plist ':question)
                              ':author))
          (\.dinosaur (plist-get plist ':dinosaur)))
      (list
       \.question.id
       \.question.author
       \.dinosaur))))

And evaluates to

("45581" nil nil)

let-plist essentially does depth first search IN THE BODY for all symbols that start with ..

Then let-plist--access-sexp builds plist-get access chain for the let like (\.dinosaur (plist-get plist ':dinosaur)).

Be wary of nesting calls to let-plist¹.

And finally, what is that sharp quote doing there? "\\`" will be read as \`, which is used to match the beginning of a string or buffer. I don't know why they didn't use ^ which is more understandable in my opinion: "^\\.", but this matches lines.

Conclusion

Many possibilities exist. I will probably stick with cl-destructuring-bind since it's built in, but it was a pretty fun exploration of macros which I don't do very often.

If you liked this synthesis, please econsider supporting this blog. It's because of supporters that I can dedicate my time and energy into exploring topics like this one and share them with others.

From 13 Jan 4XXX, have a nice day in 13 Jan 2XXX!

Footnotes:

One issue I've had with let-alist is when an inner let-alist tries to access properties from the outer let-alist form. The outer let-alist won't read the inner let-alist form to expand accessors. Meaning if the inner form accesses a .property.from.outside, it will be nil.

I don't know how to get around that, or if I should.

snd-selectric-mode v0.0.1 released

4025-01-09T14:09:00Z

I fixed the performance issues from selectric-mode that I wrote about in my article about ipc for sounds in emacs. Code is here: https://codeberg.org/MegaJ/snd-selectric-mode.

Demo (turn sound on!):

To briefly recap, last time I experimented with

selectric-mode without changes
thread spawning for each sound played
mpv as a daemon, emacs sending commands through a socket

My last option was the most performant but suffered from limitations like mixing not being very practical. Now, I found an option to mix that I've tested only on linux. It relies on a binary called snd, which may be packaged by your package manager.

When you have snd, here's a use-package incantation to get my emacs package if you use straight.el:

(use-package snd-selectric-mode
  :straight (:type git :host codeberg
                   :repo "megaj/snd-selectric-mode"
                   :files ("*.el" "*.wav"))
  ;; I recommend leaving the verbosity setting on in case you run into bugs
  ;; the first time you use it.
  ;; You can check the *snd-selectric-scheme* buffer for output.
  ;; :custom (snd-selectric-verbose-process-p nil)
  )

How it works

Ultimately, it's just emacs sending scheme commands through a comint buffer. There's nothing fancy here¹. All the heavy lifting is done by snd.

snd

snd is an audio editor, but it handles realtime audio playback and mixes it all down. It accepts commands through a repl directly, or even through a socket. It's what makes this all possible. And it runs scheme!

Not only does it run scheme, but it's the emacs of sound editing–it seems most things about the currently running program can be changed while it's running.

It was not made for my use case, but it's flexible. I've already thought about my next project with it. I want to use it like how people use fmod for games.

It does come with some drawbacks though. It lacks a really good debugger as admitted by the author of snd. It also doesn't have a community outside of a mailing list.

Performance

This is comparable to using mpv. Maybe slightly more CPU usage, but I get mixing and a bunch more functionality.

Figure 1: 20% cpu using snd

Enjoy!

This package hasn't had users other than myself yet. Please raise issues you find on https://codeberg.org/MegaJ/snd-selectric-mode/issues.

And if you have any tips on writing s7 scheme or know a lot about snd, send me word at jwow [at] 17070415 [dot] xyz.

Lastly if you enjoy my articles or any software I wrote, please consider supporting my work.

Footnotes:

the only contrived thing was to suppress the output since the comint buffer acts like a repl. There is no way to suppress the output, but I've made it possible for users to (setq snd-selectric-command "snd -b > /dev/null") which eliminates all terminal output.

I believe I can write some scheme to change how the repl prints, but I'm not sure. I actually don't know how scheme works, only that it's a compartively much smaller language spec than common lisp.

Flameshot + Emacs + Dbus

4025-01-07T12:13:00Z

I've been looking for faster ways to include screenshots and other media in org mode.

With 0 keystrokes in emacs:

The video sequence:

I press the Print Screen key, bound on my desktop to call flameshot gui.
I confirm the area selection looks good to me with Ctrl + C.
Flameshot sends a method call over dbus after it records the bytes of the image into the system clipboard.
Emacs has been monitoring for that specific method call, and (yank-media) is called automatically in the (current-buffer).
Due to a quirk in the clipboard not being ready when this notification fires, (yank-media) thinks we're pasting plain text, and fails.
The quirk is a nonissue if existing image data exists in the clipboard, so the second time I do steps 1 - 4, it works, with the newest image.
Finally I call org-toggle-inline-images (ok, 1 keystroke, but it was for the demo).

So there's a quirk, but I hope the folks at flameshot will continue to iterate on it. Or maybe it's an issue with Wayland + Gnome interaction?

Jump to Code to steal this.

Update 2025-01-09 I find that flameshot is really finicky if it will work outside of being called within Gnome terminal on wayland. Sometimes it works if I restart my computer.

Desired Workflow

Take a screenshot
Emacs should immediately either
1. add a link to the media in kill-ring so I can decide what to do with it later.
2. auto insert into the org doc in (current-buffer), allowing me to take a bunch of screenshots and then I can just rearrange them as I like, like a scrapbook.

In the video, you see me take option 2. But hopefully after reading this, you'll realize option 1 is a piece of cake too.

Tradeoffs Made

Other Approaches

Other non-dbus options exist, and they look complicated.

https://github.com/bburns/clipmon polls the system clipboard.
- Works for text and images in system clipboard.
- Only works on X.
https://github.com/cdown/clipnotify is properly evented and I experimented with calling dbus commands to send signals to emacs
- It's not as easy to package a solution using it for various reasons¹.
- Also only works on X.
- Works for text and images in system clipboard.
Use the elisp#File Notifications api, by listening to changes in the file system. For this, configure the screenshot tool to always save to a file.
- Probably the most robust, cross platform solution. I did not choose this, partially because of fashion.
- I would have to save a file first, before likely moving it.

DBUS + Flameshot approach

You might be wondering why I didn't use a DBUS event for when the system clipboard changes. It's because I couldn't find one.

Pros:

Should work on Wayland and X.
Is event-based, no polling required
Uses native emacs DBUS API
No integration layer to set up in Flameshot–it has native dbus support.

Cons:

Has 2 hard dependencies: dbus and flameshot.
Doesn't work with text, I've only supported images since I'm focusing on flameshot.
DBUS support in Flameshot could be improved.

So I've settled on flameshot

flameshot --version

Flameshot v12.1.0 (-)
Compiled with Qt 5.15.14

If you're on wayland, there's scattered help online, but see if https://flameshot.org/docs/guide/wayland helps.

Flameshot settings

I like the "Use last region" option, which you can access by running

flameshot config

Then running

flameshot gui

brings up a screen where you can select your region. (Shown in demo video). Hover over icons to learn the shortcuts.

One thing I love in emacs is browsing man pages with (woman), where you can learn more about flameshot.

(woman "flameshot")

Figure 2: animation of scrolling through woman in emacs

Code

There are only two components:

Monitor: gets notified of events in dbus. You specify fields to filter only the events you want.
Handler: handles the event. Call arbitrary code.

(defun jwow/dbus-flameshot-handler (&rest _args)
  "A dbus event handler that adds picture attachments. Forces a
 selection of media type as \"image/png\" to speed up workflow"
  (let ((completing-read-function
         (lambda (&rest _)
           ;; customize a variable for this instead to your liking
           "image/png")))               
    (yank-media))
  (insert "\n"))

;; Save the dbus object, which is a monitor, so we can unregister it if needed
;; This invocation can be derived by watching dbus-monito messages
(setq jwow/dbus-flameshot-monitor
      (dbus-register-monitor
       :session                       ; bus
       #'jwow/dbus-flameshot-handler
       :sender "org.flameshot.Flameshot"
       :path "/org/freedesktop/Notifications"
       :interface "org.freedesktop.Notifications"
       :member "Notify"))

;; To disable the auto yank-media behavior, unregister the monitor
(dbus-unregister-object jwow/dbus-flameshot-monitor)

Quirks

Notify?

Yes, I waited for Notify to get called, which populates system notifications:

I did this in part because there weren't many identifiable events from flameshot. I was a little sad about that.

Listing 1: message of flameshot using Notify after writing binary data

method call time=1736280897.494725 sender=:1.229 -> destination=org.freedesktop.Notifications serial=111
   path=/org/freedesktop/Notifications; interface=org.freedesktop.Notifications; member=Notify string "flameshot" uint32
   0 string "flameshot" string "Flameshot Info" string "Capture saved to clipboard." array [ ] array [ ] int32 5000

No Signals?

At this point, it's a bit weird IMO, that Flameshot doesn't have any signals (for broadcasting) in its DBUS interface. I think it should. That way, emacs can register itself as a proper service and get notified that way.

Emacs registering a monitor might be hacky (I don't know, I'm not a heavy dbus user) and I cannot comment on its performance. Creating a monitor is based on dbus-monitor, a shell command (and elisp command).

Figure 4: In D-Spy, I inspect the interface for Flameshot. It has 3 methods and 0 signals.

Flameshot sends an incorrect DBUS message?

In the shell command I get something like this:

method call time=1736280897.456930 sender=:1.460 -> destination=org.flameshot.Flameshot serial=9 path=/;
   interface=(null); member=attachScreenshotToClipboard array of bytes [ 00 00 00 01 89 50 4e 47 0d 0a 1a 0a 00 00 00 0d
   49 48 44 52 00 00 02 90 00 00 01 ec 0
 ...

In emacs's code path for dbus events:

dbus-event-bus-name: D-Bus error: "Not a valid D-Bus event", (dbus-event
 :session-private 1 9 ":1.576" "org.flameshot.Flameshot" "/" nil "
attachScreenshotToClipboard" jwow/dbus-flameshot-handler ...)

The interfaces is missing; seems like a bug. In any case, it doesn't avoid the issues of text being in the clipboard, and (yank-media) having no idea how to handle it.

Emacs `(dbus-monitor)`, funnels all events

If you set up a monitor, unfortunately calling (dbus-monitor) will make your previously defined handler handle all events.

I think this is a flaw in the emacs dbus api. dbus-monitor should call a built-in handler.

At this point, unregister the dbus monitor to get back to normal.

(dbus-unregister-object jwow/dbus-flameshot-monitor)

Explore More

See dbus#Top for an overview of the dbus api.
Ian Eure's informative and concise presentation at https://emacsconf.org/2022/talks/dbus/. He shows off an interface where he uses DBUS to detect if a usb drive connects to the computer.
Think about things you could do with the handler:
- Save the data to a file and save the file name to kill-ring
- experiment with (gui-get-selection 'CLIPBOARD 'TARGETS)
- pipe the screenshot to a program to resize it
- get auto-ocr, and yank the text from the image to kill-ring

Thanks for supporting my work. Turning into an unstoppable space god wouldn't be possible without your help!

Footnotes:

For one, you have to create a loop in bash to keep it listening. Which isn't bad. It notifies whenever text is selected, not even copied, as well. To change that, you have to change the C code and recompile the binary.

See https://unix.stackexchange.com/a/723818

Selective org inline image toggle

4025-01-07T06:52:00Z

Today while working on my flameshot article, I wanted to toggle inline images only where my cursor was. I was piling a ton of images and didn't want to wade through each of them.

Figure 1: animation of toggling inline images at point

Here's the same image in html that I toggle on org mode:

Figure 2: An image for toggling 1

Figure 3: An image for toggling 2

Luckily, org-toggle-inline-images allows you to specify the beginning and end of a region:

(org-toggle-inline-images t 379 448)

And here's the code:

(defun jwow/toggle-image ()
  "Toggle viewing the image from an org LINK at POINT if the LINK paths to
an image. Otherwise, toggle all images.

If the link is to an image but it does not exist, nothing will happen."
  (interactive)
  (if-let ((link-at-point (org-in-regexp org-link-bracket-re 1)))
      (if-let* ((link-context (org-element-context))
                (local-file? (member (org-element-property :type link-context)
                                     '("attachment" "file")))
                (image? (org-file-image-p (org-element-property :path link-context))))
          (org-toggle-inline-images t (car link-at-point) (cdr link-at-point))
        (org-toggle-inline-images t))
    (org-toggle-inline-images t)))

(define-key org-mode-map (kbd "C-t") #'jwow/toggle-image)

Most of the time taken to code was actually digging into ol.el and org.el to determine if something was a valid image link or not.

Hope this helped!

Org Attachment Investigation: Forced Absolute Paths on Export

4024-12-30T02:14:00Z

In this post I investigate how org attachments links behave on export. Mainly I care about how it renders html links, forcing absolute paths. I found it surprising, and abnormal.

I'm running emacs 30.0.50, an edge build, and I ran into a situation where

attachment:clipboard-20241230T022004.png

exports this <img> html tag:

<img src="file:///home/furaro/src/my-site/src/content/data/7d/167a0f-5ae4-4f45-bd29-62ec6e464173/clipboard-20241230T022004.png" alt="clipboard-20241230T022004.png">

But in html, your browser can't render that image. The link is broken and you would instead see the alt text "clipboard-20241230T022004.png"

What we'd really like is just to change the src attribute to something like

"data/7d/167a0f-5ae4-4f45-bd29-62ec6e464173/clipboard-20241230T022004.png"

So that upon exporting, the html file can look through the same path to find the image. I have not customized org-attach. Others online say they get relative paths–and they struggled to get absolute paths (link).

For this article, I consulted https://orgmode.org/manual/Advanced-Export-Configuration.html a lot.

Jump to tl;dr for the reasons.

Org Vocabulary

backend: this is the format org is being told to export to. Examples are html, odt, latex, and many others. Check all your loaded backends in org-export-registered-backends
AST: abstract syntax tree. This is a representation of an org document that's easy for lisp to manipulate. It's a tree datastructure.
parsing: this is the process of building the AST. Loosely, org copies your org buffer into a temporary buffer and certain actions like macro expansion, #+includes, and comment removal happen on this temporary buffer.
transcoder: a translater. It is a function that takes some element inside org mode (like a source block), and transforms it into a string that can be parsed by the backend.
derived backend: is a backend that has a parent. You specify transcoders when creating a custom backend for export, falling back to the parent's transcoder, for say, a link. When deriving a backend, one important keyword property is :translate-alist, which is a list of the transcoders you specify.
org-link-parameters: This is a list of link types and their behavior when certain actions are taken. Most commonly, you :follow a link, and the follow function is then used. In our case, we mostly will care about the :export. Read the help doc for the many more parameters.
Here's an example of a link type for [[nov:a-path-i-made-up]]
```
(("nov" :follow nov-org-link-follow :store nov-org-link-store)...)
```
filters: are functions. They run after the transcoder is run and get the transcoded string. The positional arguments for a filter are (text backend info), where info is a giant context object that gets populated during parsing.

tl;dr

org-attach.el bypasses transcoders and the org-link-parameter :export functionality. Why it does this, I don't know.

How it does that is interesting:

;; This is the last line of code in org-attach.el
(add-hook 'org-export-before-parsing-functions 'org-attach-expand-links)

org-attach.el decides to rewrite the temporary buffer before the AST is even parsed.

When AST parsing happens, instead of

attachment:clipboard-20241230T000706.png

we get

file:/home/furaro/src/temp/data/80/943450-583e-4236-a2d1-1e926fbb15bd/clipboard-20241230T000706.png

Thus even the link type (attachment) is lost, and replaced with file.

In depth

Let's look at why certain solutions don't work. For that, let's see what the help says:

org-export-before-parsing-functions is a variable defined in ox.el.

Value
(org-attach-expand-links)

Documentation
Abnormal hook run before parsing an export buffer.

This is run after include keywords and macros have been expanded
and Babel code blocks executed, on a copy of the original buffer
being exported.

And we run org-attach-expand-links, replacing the link with

(concat "file:" (org-attach-expand file))

And org-attach-expand always expands to a full file path.

(defun org-attach-expand (file)
  "Return the full path to the current entry's attachment file FILE.
Basically, this adds the path to the attachment directory."
  (expand-file-name file (org-attach-dir)))

The code execution cannot be influenced by a customization. Sigh.

To recap: org-attach.el rewrites the temporary buffer that org uses, BEFORE parsing the temp buffer into its abstract syntax tree format. The transcoders run after the AST is built, explaining why methods that involve the transcoding phase, do not work. This means that even creating your own derived backend alone will work. Not unless you do something to influence the behavior of org-export-before-parsing-functions.

Non solutions:

For this puzzle, these are the most likely approaches I think readers will attempt, which, at least as I've coded them, won't work.

Don't use attachments
This is fine, you can just use file: style links but that defeats the purpose of the investigation for people who rely on org attachments.
Setting an :export function for the attachment
```
(defun my-fun (path desc backend info)
  (message (concat "This is the link path: " path))
  ;; now do stuff to return the html string
  )

;; Cannot snag on the debugger, message never happens
(org-link-set-parameters "attachment" :export #'my-fun)
```
If this function ran, path would be "clipboard-20241230T022004.png", but this never triggers. This would be a rather clean solution as this specifically triggers only for attachment: links. The org export machinery's backend should detect links with custom protocols using the built-in org-export-custom-protocol-maybe function. ox-html does, for example.

One caveat of this method is that you want to return a string specific to whatever the backend target is (latex, html, markdown, etc.) This is a hit for maintainability.
Defining a custom transcoder (maybe in your own derived backend)
for something like (link . jwow/org-attach-link)
The AST gets passed to ox-html's org-html-link, but the :path will have been populated as an absolute path, of course because the input for the AST has already been changed.

You can verify in org-html-link, calling this line from edebug:
```
(org-element-property :path link)
```
You'll be able to detect the long UUID that's characteristic of an org attachment, but this is hacky.

Operating after the transcoder, hooking into the org-export-filter-link-functions mechanism.
Again, like the last approach, you have to guess from the full file path that this WAS an attachment, then most likely do a string replace operation.

(defun my-org-link-filter (text backend info)
  (message text)
  ;; do something to text
  )

(add-to-list 'org-export-filter-link-functions #'my-org-link-filter)

;; Text in this case is
;; <img src="file:///home/furaro/src/temp/data/80/943450-583e-4236-a2d1-1e926fbb15bd/clipboard-20241230T000706.png" alt="clipboard-20241230T000706.png">

It is fair to note that deciding on a custom string/directory for org attach to use can work, and you can regexp replace, with that coupling of systems.

One solution

I thought a solution was best found by removing this behavior of expanding links before parsing:

(remove-hook 'org-export-before-parsing-functions 'org-attach-expand-links)

After that, a lot of reasonable approaches are possible.

But let me tell you why I decided against that.

I didn't want to maintain different :export backend targets, and I wasn't interested in tracing through how org decided to generate the html for attachments, in fact, I liked how images are just handled by probably one of the most used org-links. Can we hook into machinery for that?

This was likely what led to the design decision in org-attach.el–they didn't want to deal with a new link type having to bloat up the codebase to support, html, md, latex, odt, and others. Even the docs say attachments should behave like files.

I just wish there were a customization setting to specify whether the link should appear relative or absolute.

So all I did was change org-attach-expand after all:

(defun jwow/org-attach-expand (file)
  "Return an expanded relative file path for a FILE where
  links look like [[attachment:FILE]], relative to the org file
The return string may start with data/0a/39..."
  (let* ((attachment-abs-path (expand-file-name file (org-attach-dir)))
         (org-file (buffer-file-name
                    (org-element-property :buffer (org-element-context))))
         (org-file-dir (file-name-directory org-file)))
    (file-relative-name attachment-abs-path org-file-dir)))

Season to taste.

One great thing is that C-c C-o remains functional. This way, all exporters that already handle file links should continue working with your relative path.

UPDATE 2025-01-07: I've realized that this breaks org-display-inline-images, so be careful about that¹. The other options we discussed, like an export filter, would still work.

It works!

If it works here's an image of this section before I added this sentence:

I haven't tested this extensively, but it seems to work for now.

Explore more

Lastly, if you like what I'm doing, please consider sponsoring me.

John Kitchin wrote an article on easier facilities to extend org links (link)
Tony Aldon wrote a piece on links (also on reddit)

Footnotes:

Which is odd in terms of its design, because org-display-inline-images has a check

Listing 1: Found in /usr/share/emacs/31.0.50/lisp/org/org.el

(and file (file-exists-p file))

after the variable file is bound. And relatives paths pass the file-exists-p check.

Rewriting org links on export

4024-12-29T06:56:00Z

Skip to Solutions if you only care about approaches to rewriting org links on export. I recommend referring to Org Vocabulary if you're not sure what I'm talking about.

My images weren't linking properly so I had put off including images in my posts. Here's an image of my org doc:

Here's a small conundrum dear reader:

Problem

Given this file link on org mode:

[[file:../assets/media/changing-org-html-export,-plain-list/2024-12-19_12-31-41_screenshot.png]]

How would you change the path on html export to the following:

../../assets/media/changing-org-html-export,-plain-list/2024-12-19_12-31-41_screenshot.png

This is what we want the <img> tag's src to be. However, what we get is predictably:

../assets/media/changing-org-html-export,-plain-list/2024-12-19_12-31-41_screenshot.png

Context: A Tale of Two Directories

The blog layout I have isn't completely symmetric with the source files that generate the blog, because I don't generate a <post-name>.html, I have instead <post-name>/index.html, meaning the relative paths must go up one more time.

Here's a concrete example of a post with an image:

~/src/my-site/public/blog/changing-org-html-export,-plain-list/index.html

And the image is located in

~/src/my-site/public/assets/media/changing-org-html-export,-plain-list/2024-12-19_12-31-41_screenshot.png

Files in public/ are what get uploaded.

So from a particular blog page (like this one), we would need to climb up twice (../../), which brings us to public, then we can access all those racy images of code.

So dear reader, how would you approach this?

Skip to Solutions. I've included some vocabulary to facilitate discussion about org internals.

Org Vocabulary

backend: this is the format org is being told to export to. Examples are html, odt, latex, and many others. Check all your loaded backends in org-export-registered-backends
AST: abstract syntax tree. This is a representation of an org document that's easy for lisp to manipulate. It's a tree datastructure.
parsing: this is the process of building the AST. Loosely, org copies your org buffer into a temporary buffer and certain actions like macro expansion, #+includes, and comment removal happen on this temporary buffer.
transcoder: a translater. It is a function that takes some element inside org mode (like a source block), and transforms it into a string that can be parsed by the backend.
derived backend: is a backend that has a parent. You specify transcoders when creating a custom backend for export, falling back to the parent's transcoder, for say, a link. When deriving a backend, one important keyword property is :translate-alist, which is a list of the transcoders you specify.
org-link-parameters: This is a list of link types and their behavior when certain actions are taken. Most commonly, you :follow a link, and the follow function is then used. In our case, we mostly will care about the :export. Read the help doc for the many more parameters.
Here's an example of a link type for [[nov:a-path-i-made-up]]
```
(("nov" :follow nov-org-link-follow :store nov-org-link-store)...)
```
filters: are functions. They run after the transcoder is run and get the transcoded string. The positional arguments for a filter are (text backend info), where info is a giant context object that gets populated during parsing.

Solutions

Solution 1

postname/index.html -> postname.html

We can simply make a file, <postname>.html and lose the directory above it. Then we have symmetry. But funnily enough, I didn't realize this until the end that my files were actually named index.html¹, and that I had an asymmetric directory structure. Instead, I was trying so hard to get Solution 2 to work. Plus, I use 11ty to place these files and I didn't want to dig into it.

Since writing the outline of this article I have actually chosen this solution².

Still, the other solutions are worthy considerations to have at your arsenal if you are a nerd, or you just deal with exporting to different targets arbitrarily.

Solution 2

../assets -> /assets

I think assets should be located from the root serving directory, so I'd prefer a prefix of /assets/. In the org document, ../assets must remain so we can follow the link on our filesystem using C-c C-o.

What about a custom transcoder, where we modify the link to be an absolute path, then pass it into org-html-link, which translates those file links in org mode to html?

This didn't work–at least using this idea alone doesn't.

The problem when you do that is inside org-html-link, there is a call to org-export-file-uri, which will transform a path's prefix into

../assets -> file:///assets

When looking up a file://, the browser will attempt to look up something on the user's computer (and likely the browser will stop that from happening), not the image from the web server.

At this point, I gave up on absolute pathing from the webserver because messing with too much org machinery won't be maintainable. See Solution 5 for a different way to do this.

Solution 3

../assets -> ../../assets

After trying to make solution 2 work, I had edebugged org-html-link and org-publish-file-relative-name and knew how to rewrite the paths. I just needed to make sure the machinery wouldn't do something unexpected.

Working from Solution 2, I rewrote the export path using pretty much the same code, the only difference was how to handle string replacement.

(defun jwow/org-html-link (link desc info)
  "Transcode a LINK node from an Org AST to HTML.
DESC is the description of the link and can be empty
INFO is a giant context object of the export that's decorated with a ton
of data, and metadata with things like the buffer name, input file, etc."
        (let ((type (org-element-property :type link))
              (raw-path (org-element-property :path link)))
    ;; rewrite ../assets -> ../../assets
    (if (and (string= "file" type)
             (string-prefix-p "../" raw-path))
        (let ((modified-path (replace-regexp-in-string
                              ;; I used re-builder to come up with this regexp
                              "\\(\\.\\./\\)+assets"
                              "\\1../assets"
                              raw-path)))
          (org-html-link (org-element-put-property link :path modified-path)
                         desc info))
      (org-html-link link desc info))))

The key bit is the replace-regexp-in-string, where we simply tack on one more ../. And it worked!³

Solution 4

If you read my article on investigating attachment links, you'll also know that we could vary Solution 2 and 3, but slap on an export filter instead.

Specifically, add a function to org-export-filter-link-functions to regexp replace the strings and make them relative.

This doesn't feel right to me because of the indirection–the org object has already been made into html.

For this approach, you'd care about not polluting the export functionality in other projects. For this you should use a .dir-local, or file local variable depending on how big your project is, and how often you need this link export code path to happen. See my article on .dir-locals for more.

You can also ensure the filter happens ONLY in a file⁴ using a function you define in an org src-block.

Solution 5

Use an #+ATTR_HTML:

#+ATTR_HTML: :src /assets/media/changing-org-html-export,-plain-list/2024-12-29_04-43-19_screenshot.png
[[file:../assets/media/changing-org-html-export,-plain-list/2024-12-29_04-43-19_screenshot.png]]

from isamert on stackoverflow

This is the img I've exported:

Check the img src attribute, it's absolute!

Solution 6

Create a custom link type for

../assets -> ../../assets

This solution comes from the revered Kris Jenkins mentioned in the same stackoverflow post as isamert.

For this solution you would no longer use the file type, instead, something like img in your markup: [[img:../assets/img.png]]

For this problem, one could write

(org-add-link-type
 "img"
 'org-custom-link-img-follow
 'org-custom-link-img-export)

;; omitted the follow definition

(defun org-custom-link-img-export (path desc backend info)
  (replace-regexp-in-string
   "\\(\\.\\./\\)+assets"
   "\\1../assets"
   path))

This defines only follow and export functions. Look into (org-link-set-parameters TYPE &rest PARAMETERS) if you want to further define link behavior, especially if you care about using org-store-link capabilities.

One drawback is you might need to change a bunch of your org links from file -> img. This can be done by matching on an image suffix on files. Further, you have to implement 3 functions for export, store, and follow.

Conclusion

There is a surprising amount of logic in links. At this time, I have other broken features in my edge build, reproducible with emacs -Q.

In the future I'll most likely need to work on linking between different articles as well.

The org element api is really powerful, and learning the ins and outs of a tool you use for hours every day can help you build a more flexible, custom workflow for spreading conspiracy theories. I recommend you try it.

If this post helped you, please consider sponsoring me or helping me find work in software development or adjacent fields. Your help allows me to walk into grocery stores as a proud emacs user.

Footnotes:

Hey, I started this blog on December of Earth Year 2023, then…well a lot of things happened and I forgot how I set up this project.

modifying my 11ty build was a bit painful to do this. Check out https://github.com/11ty/eleventy/issues/584.

Readers should make note that this involves also swapping out the trancoder to use jwow/org-html-link. In this case once you realize this works, you'd likely create a new derived backend from html. See the org manual for examples.

⁴

The #+BIND toplevel keyword, details in the org manual. It might be possible to define different BINDs in property drawers on subtrees too, so you can super localize how links in one subtree are exported if you need to.

You Want To Modify Exported Footnotes In Org Mode

4024-12-29T01:13:00Z

Today I had been struggling with org's footnotes, and I'd like to share how I've done them up on this site.

I wanted them to look more like in Karthink's excellent blog.

And now I'm satisfied enough, though my html structure is different.

The info manual doesn't list out the options in one place¹, so I had to. Further, a lot of the names were so similar as to be confusing.

I didn't override org-html-footnote-section, but others have.

org-html-footnotes-section is format string of html that declares an header (by default an <h2>), and prominently displays a visible "Footnotes" on default settings.
- By default it is set to:
```
<div id="footnotes">
    <h2 class="footnotes">%s:
    </h2>
    <div id="text-footnotes">
        %s
    </div>
</div>
```
org-footnote-section, a string (mine is the string "Footnotes"), will substitute for the first %s, in the format string org-html-footnotes-section.
The second %s will be individual footnote items.

org-html-footnote-section, a function, is what controls the html structure that wraps around each footnote definition.

it is responsible for the hardcoded footnum, footdef, and footpara css classes, and the placement of divs and <a>s. There is no built in facility to change the structure:

(contents (org-trim (org-export-data def info))))
               (format "<div class=\"footdef\">%s %s</div>\n"
                       (format (plist-get info :html-footnote-format) anchor)
                       (format "<div class=\"footpara\" role=\"doc-footnote\">%s</div>"
                               (if (not inline?) contents
                                 (format "<p class=\"footpara\">%s</p>"
                                         contents))))))))

org-export-collect-footnote-definitions, a function, collects footnote definitions from the rest of the org doc. It is only called by exporters in the mainline codebase, like by org-html-footnote-section. You shouldn't need to touch this unless you're doing something fancy, or want to put footnotes in custom places.
org-html-footnote-format, a string, is by default "<sup>%s</sup>", this is the html used for listing your wrapping the numbered anchor tags in the footnote section.
org-html-footnote-separator, a string, is what separates your footnotes in the main body of your text. Mine just puts a ","

From the image at the top of the post, I have

#text-Footnotes.footdef

then sup and .footpara appear for every footnote.

The CSS, I modified from this post. The key issue was that my posts have a footer, and thus I needed the well-known CSS clearfix hack.

div.footdef::after {
    content: "";
    clear: both;
    display: table;
}

.footpara {
    width: 99%;
    text-align: left;
    float: right;
    padding: 0;
    margin: 0;
}

.footpara {
    display: inline-block;
    font-size: 1.1rem;
}

.footdef  {
    margin-bottom: 0.5rem; 
    line-height: 2em;
}

If you disable any rule in div.footdef::after , you'll see what I mean:

I was tired and confused about all the variable names when I first did this, but I'll probably reach for flexbox or CSS grid next time. There's no reason to use floats with these better options.

Alternatively, I could have just used different transcoders for the footnotes, like pdixon.

Footnotes:

I just wanted to have a footnote in this post.

org#Creating Footnotes mentions how you create them, disable them, and minor notes on some display features.

org#CSS support mentions CSS styling. And that's about it in the manual.

The rest was gathered through looking at help docs.

Bookmark that rg

4024-12-25T07:12:00Z

I like bookmarks, don't you? But something missing was an ability to save a bookmark for an rg.el search¹^,².

Here is my solution after digging into the rg lib:

Code

Define a function to serialize the bookmark
To find your way back, you'll need to record information. Bookmarks will jam all your data into a big list.

What data to record? Here, I had to search for local variables that mached ^rg- . Check out using (buffer-local-variables) (originally I did this the slow way of using C-h v). Then I realized it was a struct, and traced through the code to find out what function to be called (rg-run) and record data to pass into to it:

(defun rg-bookmark-make-record ()
  "Make a bookmark record for the current rg buffer."
  `(,(format "rg: \"%s\" ∋ %s ∀ %s"
             (rg-search-pattern rg-cur-search)
             (rg-search-dir rg-cur-search)
             (rg-search-files rg-cur-search))
    ;; pattern is a synonym of query in rg.el
    (pattern . ,(rg-search-pattern rg-cur-search))
    (files . ,(rg-search-files rg-cur-search))
    (dir . ,(rg-search-dir rg-cur-search))
    (literal . ,(rg-search-literal rg-cur-search))
    ;; Found some state in the source code; 
    ;; I don't think these are needed, but they are present 
    ;; (search-cfg ,(rg-set-search-defaults (cdr body)))
    ;; (local-bindings (rg-search-parse-local-bindings search-cfg))
    ;; (iargs (rg-search-parse-interactive-args search-cfg))
    (flags . ,(rg-search-flags rg-cur-search))
    (handler . rg-bookmark-handler)))

Now we need a way to read that stored data and jump to the bookmark

(defun rg-bookmark-handler (record)
  "Jump to a bookmark's url with bookmarked location."
  (let ((query (bookmark-prop-get record 'pattern))
        (files (bookmark-prop-get record 'files))
        (dir (bookmark-prop-get record 'dir))
        (literal (bookmark-prop-get record 'literal))
        (flags (bookmark-prop-get record 'flags)))
    (rg-run query files dir literal nil flags)))

We could shorten this code since there's so much repitition, but let's keep the simplicity for now.

And finally, bookmark-make-record-function must be overridden in the buffer-local vars for the mode:

(defun rg-set-bookmark-handler ()
  "Assigns `bookmark-make-record-function' to a custom function."
  (set (make-local-variable 'bookmark-make-record-function)
       #'rg-bookmark-make-record))

(add-hook 'rg-mode-hook #'rg-set-bookmark-handler)

Test

I activate a rg search for the regexp version: draft and tweak a flag.
I M-x bookmark-set.

The prompt shows

Set bookmark named (default rg: "version: draft" ∋ /home/refurb/src/my-site/ ∀ org):

I press <Enter>

I inspected the saved bookmark:

("rg: \"version: draft\" ∋ /home/refurb/src/my-site/src/content/ ∀ org"
 (pattern . "version: draft")
 (files . "org")
 (dir . "/home/refurb/src/my-site/src/content/")
 (literal)
 (flags)
 (handler . rg-bookmark-handler))

Close the ripgrep buffer.
Using M-x bookmark-jump, I can jump and see the ripgrep search I just saved, the buffer loads up, I see my results!

Conclusion

Now you've got another person (me) telling you how great bookmarks are in emacs. Give it a shot and start making your own bookmark types!

I did not cover making a "category" of bookmarks. Drew covers it here on emacs.stackexchange.

The search hits were sparse when I looked for what this use case of bookmarks. Maybe I'm alone out here.

If this post looks similar to one by overvale, it's because I read one of their articles: https://www.overvale.com/emacs/extending-emacs-bookmarks.html

Merry Christmas!

Full code

Maybe I should package this up?

(defun rg-bookmark-make-record ()
  "Make a bookmark record for the current rg buffer."
  `(,(format "rg: \"%s\" ∋ %s ∀ %s"
             (rg-search-pattern rg-cur-search)
             (rg-search-dir rg-cur-search)
             (rg-search-files rg-cur-search))
    ;; pattern is a synonym of query in rg.el
    (pattern . ,(rg-search-pattern rg-cur-search))
    (files . ,(rg-search-files rg-cur-search))
    (dir . ,(rg-search-dir rg-cur-search))
    (literal . ,(rg-search-literal rg-cur-search))
    ;; Found some state in the source code; 
    ;; I don't think these are needed, but they are present 
    ;; (search-cfg ,(rg-set-search-defaults (cdr body)))
    ;; (local-bindings (rg-search-parse-local-bindings search-cfg))
    ;; (iargs (rg-search-parse-interactive-args search-cfg))
    (flags . ,(rg-search-flags rg-cur-search))
    (handler . rg-bookmark-handler)))

(defun rg-bookmark-handler (record)
  "Jump to a bookmark's url with bookmarked location."
  (let ((query (bookmark-prop-get record 'pattern))
        (files (bookmark-prop-get record 'files))
        (dir (bookmark-prop-get record 'dir))
        (literal (bookmark-prop-get record 'literal))
        (flags (bookmark-prop-get record 'flags)))
    (rg-run query files dir literal nil flags)))

(defun rg-set-bookmark-handler ()
  "Assigns `bookmark-make-record-function' to a custom function."
  (set (make-local-variable 'bookmark-make-record-function)
       #'rg-bookmark-make-record))

(add-hook 'rg-mode-hook #'rg-set-bookmark-handler)

Footnotes:

In my opinion, rg.el doesn't have good support for persisting saved searches (nor should that be its focus).

Update 2024-12-28, burly.el can bookmark rg, but I've found it doesn't restore window configuration properly with rg buffers.

4024-12 In Review

4024-12-23T21:37:00Z

My meaningful achievements this month:

I completed 16 blog articles. Which is 4 months worth of blog posts. More are on the way, because I actually worked on 21 blog posts.
Solved technical problems for this blog, people can now experience audio, video, pictures.
Wrote a letter of recommendation for a student.
Decided to not be on nixOS–it's wonderful to be back on manjaro for developing.
Did 1 unit of Duolingo per day. The time it takes is around 2-2.5h/day now. Noise disrupts me greatly.
- Edit: I got knocked out of diamond league; I prioritized writing blog posts the last week of the year.
Daily leetcode. Got my 2024 annual badge.
Volunteered at a blood donation place, and it's also good I realized that particular place did not need me.
Began volunteering at nonprofits. Technically accepted a position at 2 of them so far. Another is interested.
- I have been so busy with debugging things that I haven't given them too much time however.
- My first PR at a nonprofit got merged, second one on the way.
Coding related
- linux
  - Got a better understanding of how linux permissions work.
  - Played with dbus more.
- org
  - Learned how to use the org-mode AST, wrote a short function to call codeblocks.
  - It's so silly, but #+begin_export html was not something I realized was a thing. But now I've simplified one of my pages quite a bit. The mixture of org, nunjucks, and 11ty feels more manageable this way.
  - Learned the org export process deep enough to feel like I can wield it like a baseball bat in a china shop, then reverse time, and finely stich the structure of destroyed china back in place, with my mind.
    - Well, I'm not that good yet.
  - Shipped my template for debugging web service requests. Helped me review noweb.
  - I learned a lot about org. I mean, just check out this month's blog posts!
- php
  - Finished "PHP In Easy Steps" (Maybe I did this last month?)
  - Finished "Wordpress for Dummies 9th edition" and got a handle on wordpress. I understand how to manipulate old themes at least
  - Wrote a php package php-binary-index-tree, and it's on packagist.
    - This was very painful coming from a position where I didn't know php, but I learned the most from it
    - PHP unit is also bizarre coming from BDD frameworks where tests are organized into suites and can be nested.
  - Tried out dape, even put up an issue I was facing.
  - Set up a php dev environment for myself, to be ready to volunteer doing wordpress + php based frameworks.
    - I hooked up eglot, php-ts-mode, php-mode, web-mode, phpactor, dape, company, and xdebug.
    - I also managed to fix some issues on my own about packages bringing in conflicting dependencies (this also involved me upgrading to an edge build of emacs)
- more emacs
  - Built a working prototype of an elisp webapp (hope to finish next month). It uses elnode, which is now 10+ years old. I don't know if I want to do this over the top and add an unecessarily complex storage layer. It would be fun to practice using the emacs pg library, but completely uneccessary.
    - Finally learned sxml from esxml, and it is actually really nice to have html as s-expressions.
    - Couldn't find out from searching online how to add literal html (it kept escaping). But there is a way: *RAW-STRING*. I only found the github issue after debugging into the source code.
      - How does daviwil not use a raw-string right now in his source for systemcrafters?
  - Got used to/aware of rg.el's more advanced capabilities (the transient, setting custom searches, history).
  - Debugged a ton of elisp, fixed some of my own issues with transients surprisingly.
    - Every time I debug a tool, I feel more confident about using it.
  - Prototyped a way fill emacs with the system clipboard using dbus. It works enough, but it'd be nice if I packaged it.
    - It has issues and I want it to translate image clipboard data well.
    - It relies on clipnotify. Which is X only. But I run X on Wayland. Still not sure how this window environment stuff works.
  - Read more about state charts, which have a charm to them. Started thinking about implementing a State Charts library in elisp. I want my cursor to be in different colors depending on different mode states, and this is too complicated to deal with without state management.
  - Have export on save as a .dir-local has saved a bunch of time writing articles.
  - Creating a custom hydra to do various blog actions, like renaming files, starting a local server, quick making a new post, etc..
- Keybindings
  - Retooled some keybindings; discovered the joy of project.el and how project-switch-project can save me time, as well as helm-projectile-find-file (I use a mix of helm and simple completing-read).
    - I usually have a mass of unrelated files with similar file names that makes completing more difficult.

Meaningful losses:

Not eating enough. Not getting up early enough to buy eggs on discount.
Got sick. Heart pain.
Loss of sleep due to noise, sickness, and even excitement.
Spending less time with people than I maybe should.

What I'm grateful for:

My friend letting me stay with him
eggs
Leftovers (there were a lot).
Local public library
I actually had a Thanksgiving, and the food was really good.
I'm writing; I don't know why I'm so motivated. Maybe because it's about emacs, workflow, productivity.

Looking ahead:

More unemployment / lack of housing
Rising price of eggs
Increased health risks due to not seeing a doctor
An increased focus on trying to build engineering cultures across dimensions.
Not sure if I'll replace 11ty, I like the flexibility it gives me with tags/index pages, but I also want org file cross linking to translate properly.
I have a ton of hobby projects to release as hinted above. I want to look back, and see that I followed through.

How You Do Project Euler in Emacs Lisp

4024-12-21T05:41:00Z

Upon the late summer of 2023, I finished problems I'd missed from project euler 1-100 in emacs lisp¹ just before my life fell apart. I knew I'd be ok though, seeing as I would live 2000+ years into the future to write this blog post.

The two things I kept using repeatedly were

edebug: step through capabilities!
This tutorial by endlessparenthesis was helpful.
And the manual is seriously worth reading.
cl-loop: for the most expressive looping I've ever experienced in my life².

I didn't find anyone else who published their emacs lisp solutions up to 100. But there were of course lispers who had done far more. I was definitely inspired by a few others before I started:

someone else had a collection of a few dozen of their solutions posted as a giant file of HTML on their blog. (They were from Mexico? I don't remember)

skeeto's commmon lisp solutions, and a lot were just loop macros. He might have even remarked about this on his blog.
I also watched an emacs youtuber, who did common lisp, talk about how they loved the loop macro so much.

Anyway, cl-loop is a super power. You can read about it in the cl-lib info manual. Once you get used to the DSL, you can rip through a lot of earlier problems! It's really performant for elisp.

Here's just a function to generate a hash table from an input list:

(defun jwow/list-to-hash-table (input-list)
        (cl-loop for (k v) in input-list
                                         with ht = (make-hash-table :test #'equal)
                                         do (puthash k v ht)
                                         finally return ht))

I find this more expressive and faster to grasp than using a mapping function with a lambda:

(defun yaml--alist-to-hash-table (l)
  ;; ommitted...
  (let ((h (make-hash-table)))
    (seq-map (lambda (cpair)
               (let* ((k (car cpair))
                      (v (alist-get k l)))
                 (puthash k v h)))
             l)
    h))

Footnotes:

The notebook is too messy to be of easy value https://codeberg.org/MegaJ/euler-elisp. It's also not very readble unless you're opening up my notebook in org mode. Still, I consider it a major achievement! Might brush it up to make it more comprehensible some day.

I wonder if this article is just an ad for cl-loop, it's just so good! Maybe this should be a short series where I actually show some solutions to project euler.

Org Babel Evaluation From Afar

4024-12-21T04:54:00Z

I often write many src-blocks; I prototype a lot of things. And I love seeing what my code spits out a lot.

tl;dr: I built an auto-completion to evaluate source blocks in an org buffer.

Figure 1: animation of executing a block when point is not on block

In an hour's time of prototyping I can't avoid having a bunch of blocks in a big, structured mess. In fact, this blog post was originally just a sandbox for trying to build a completing-read interface to evalaute src-blocks.

What I've done up till now when I'm somewhere in the buffer is

isearch for the block, be it the code in the body of the #+begin_src or the name of the block if I've done that
go through isearch candidates until I find the one
C-c C-c to evaluate the block
then pop mark until I'm back to where I was

And it's too slow! I never felt like I was able enough to find more flow in this, but then

I was reading around to try and change ox-html's rendering for my plain-list
found Xah Lee's small tutorial (more of a showcase really) on org's element API,
and today I bounced over to Org's official docs for Org Element API,

and I just realized…yeah I can fix that.

I've just gotten gradually much better at elisp over time than when I started, maybe around 2015.

So here's what we'll do:

Ask org to give us an AST¹,
use AST helper methods to grab source blocks and their data
build candidates from relevant nodes of the tree
use completing-read, the minibuffer completion mechanism, to let to user select which code block to run
finally, execute the block

and we didn't need to move our cursor!

I debated whether to write this blog post to show you how to debug and figure this out on your own, but it didn't come out that way this time. That's one thing I think is lacking–those lispers don't show their process!

Inspiration

This is the piece of code that inspired me from worg. Maybe it'll inspire you in how easy it is to work with org's abstract syntax tree:

(org-element-map (org-element-parse-buffer) 'item
  (lambda (item) (eq (org-element-property :checkbox item) 'on))
  nil t)

(Don't forget to read the docs on those arguments!)²

Code

Anyway, here are the goodies.

An example block

First I need a src-block in this org document that I'll call for this tutorial:

(message "running my block")
(format "%s" "I'm currently looking for work")

The function called by `org-element-map`

I create a list of two-element lists. The first element is for display, the second element is for the point position.

(defun jwow/transform-src-block-candidates (item)
  (let* ((name (org-element-property :name item))
         (begin (org-element-property :begin item))
         (data `(,(format "%s:L%s" name (line-number-at-pos begin)) ,begin)))
    data))

Why do I want the point position? Because I plan to call org-babel-execute-src-block³. Next let's see the results from using org-element-map.

Using org-element-map

There are multiple types of source blocks; I don't use the inline ones at all, but I included them:

(org-element-map (org-element-parse-buffer) '(src-block inline-src-block)
  #'jwow/transform-src-block-candidates)

| inspiration from org element api:L46          | 2316 |
| the block I'll call:L60                       | 2709 |
| jwow/get-src-block-candidates:L72             | 3069 |
| org-element-map example:L93                   | 3991 |
| completing read example:L117                  | 4838 |
| finished code, find-src-blocks-in-buffer:L129 | 5180 |

Autocompleting candidates

I began prototyping this separately. Now I needed to show candidates that the lambda returned.

Prototyping completing read is super easy:

(format "Completed for this: %s" (completing-read "Execute block: " '(foo boo roo)))

Yields:

Org Src Run

Try running this in one of your org buffers and try to run a piece of code.

(defun jwow/org-src-run ()
      (interactive)
  (let* ((found-src-blocks  (org-element-map
                                (org-element-parse-buffer)
                                '(src-block inline-src-block)
                              #'jwow/transform-src-block-candidates))
         (chosen-block (completing-read
                        "Execute block: "
                        found-src-blocks))
         (block-start (cadr (assoc chosen-block found-src-blocks))))
    (save-excursion 
      (goto-char block-start)
      (org-babel-execute-src-block nil
                                   (org-babel-get-src-block-info)))))

Takes about a second to get completing read loadaed for a 7k line org file on my box.

More efficiently?

Of course I thought of making the code faster.

One thing that could be improved is avoid using assoc, which is a linear walk through a linked list. For repeat calls, we might be able to use a hash table mapping src-block names to point positions.

But that creates the issue of stale cache (the block position in the buffer changes as you type). We could save the AST for the block, to call functions on it–but then the code might change too.

This introduces way more complexity. And it's good enough for now.

I've learned I should spend more time communicating vs less time tweaking code.

Footnotes:

I avoid using regexp; asking org to parse is slower, but more accurate. Parsing is much better if your use case covers other elements than just src-blocks.

I didn't read what the last argument (here it's t) did, so I kept wondering why it was exiting only after listing one element. Once I removed it, I saw my lamda fire for each matching element.

Found by looking through C-h k C-c C-c to find org-ctrl-c-ctrl-c jump to source, find out some functions there, then jump to some more.

Embedding audio/video in org html export

4024-12-19T22:28:00Z

Embed whatever you want through org mode export.

Despite the title, the strategies presented work to any export format, not just html. I use the lens of exporting some media in org mode to highlight some of org's features and some nifty integrations you should consider!

Let's talk about building speed in an org workflow for inserting some markup for media assets.

You're one of those or audio/visual/writing people¹. You make content. And you use emacs. And you've heard about org mode, and you heard about generating static html files. Maybe you even make a static site using org mode like this one. All is well. You are in the right place.

You go along and discover some pretty things, some convenient things, and they taste exquisite. You start wondering about gathering more strength, more velocity, more followers, more brand deals, more estates, more politicians, more armies, more nukes, more moon colonies….

But first you want to embed <audio> in html. Or <video> for that matter. Or some custom tag. Ultimately you need it in html in some way or form. Well, you've got options:

Methods

use an #+EXPORT_HTML block.
You write your html inside it, you export.
This option works. It's a bit verbose to use each time. Even with the shorthand

@@html: my custom <audio><audio/> @@,

this is a drag. Even if you make some autocomplete for the dang thing. Really the only thing that changes is the audio file usually.

And if that's so…
use a macro that expands to content of an #+export_html block.
It's reusable, it encapsulates what changes–probably the url or filename you use to serve the audio. You set it up once, and you have clean looking org documents. Begone, clutter!
But you may not like the syntax of calling an inline macro:
```
A podcast about deep sea exploration puns: {{{ my_macro("../audio/my-podcast.mp3" }}}
```
And what's more, you have to make sure that macro is defined in every org doc you write so that you can use it. That's kind of a pain if you use the #+includes feature for this.

But is there an easy mechanism making macros available without having to write extra markup in your document each time?
Global macros
Ha! Of course there is: org-export-global-macros. I didn't go for this but it would work for me, and probably you. There aren't many downsides but you might want them on a per-project basis to make sure they don't clash. You can have a buffer local variable to set this for every org buffer of your project.

Macros are certainly flexible, reusable, and great!

But I didn't go that route. I used something more specific…something that hooks up to some other nice emacs mechanisms that we'll talk about…
Drag and drop your file
Drop a file into an org buffer and boom, the link us automatically in your org markup.

You can even customize drag and drop capability, which we don't explore right now², and generate custom markup upon dropping.

What you say, you don't use your mouse and you prefer mostly keyboard? Or maybe the file isn't local, and you dragging and dropping sounds like the aftermath of a hard drug to you…

Use a user-defined org-link, like [[audio:../audio/my-podcast.mp3]].
Systemcrafters does it that way.

Now just like any other org link, you can jump to it, specify its export behavior, and even its org-store-link functionality.

Native ones exist:

file://~/Downloads/fe3h mural tapestry.jpg
info:Org
[this blog
elisp:(message "What's the price of eggs today?")

You can click on these in org mode! Or move point over and C-c C-o to open the links.

If you want to create your own link for [[audio:../audio/my-podcast.mp3]]

;; For this example we only care about format being html
;; if you were exporting to say, md, or something else,
;; you'd have to handle that 
;; Also omitted is adding text for a11y 
;; https://developer.mozilla.org/en-US/docs/Web/HTML/Element/audio
(defun jwow/audio-link-export (path desc format)
  (format "<audio controls src=\"%s\" >
Your browser does not support the audio element.
</audio>"
          path))

(org-link-set-parameters "audio"
                         :follow #'org-link-open-as-file
                         :export #'jwow/audio-link-export)

Now

audio:../assets/media/embedding-audio-in-org-html-export/t-rex-roar.mp3

exports to this:

Your browser does not support the audio element.

This is great but it still seems I'm typing a ton of characters! All that "[[" and "audio" and the path of the file!

So let's autocomplete for the link, starting with the link type

There's a lot of different methods you can customize for auto complete. You can use a various completion framework like completing read, helm, company, corfu, counsel, and others. Or make a quick expandable template using yasnippets. To keep this section focused, let's build on a link strategy.

How do we complete for an audio link?

C-c C-l in an org buffer:

Here's the code that makes that possible:

;; I have a buffer local variable that defines my dir dynamically
;; I'm hard coding it for this demo for transparency, and so you can change it
(defvar audio-dir
  "~/src/my-site/src/assets/media/embedding-audio-in-org-html-export/")

;; Also my post is an org file inside ~/my-site/src/content/, which is 
;; important for building a relative file link

(defun jwow/org-audio-complete-link ()
  "Bring up autocomplete for audio files in a specific directory."
  (let* ((file (read-file-name "File: " audio-dir))
         (pwd-relative (file-relative-name file default-directory)))
    (concat "audio:" pwd-relative)))

(org-link-set-parameters "audio"
                         :complete #'jwow/org-audio-complete-link)

Sources say you can also complete for link abbreviations ³^,⁴, though I've had no luck with it. complete-symbol is what gets called, but I have 0 results with my cursor in front of [[. If I have [[a typed and then complete, I have an ispell error.

Completion for a non local resource
For example, if you want to autocomplete for a youtube video by asking youtube for the video ids, you're going to hook into that corporation's custom api.

We won't explore that here because every API is different, but it's entirely possible you want to embed someone else's content, and make your meta content commentary. Emacs is more than capable.

Check out https://github.com/atheriel/helm-twitch for an example of completing for twitch streams using helm. (This was also featured in an emacs conf talk).

You've succeeded
Months pass by, you gain quadrillions of followers. Like me, you become a guarded king secluded from his own dominion, cursed with an-all-too-outrageous-success with his interstellar, intertemporal blog.

Then suddenly you get an email mentioning how your content has been banned/flagged/lost in the corporate dungeons of VROnlyTubelrSoundXCloudArtBookSyAIHub. It was your most liked piece of genius on the web! Oh no!
Renaming links
You reupload material for your fans. But now the corporation gives you a new video id. Now you've got to replace all those links in org mode!

You can't bear the tedium of doing it manually. So what do you do?

If you program enough you probably have tools you go and use for this already. Since you're on org mode though, I'll point you to

M-x project-query-replace-regexp

I'd recommend tracking your project with version control then do this. You can one-shot all the replacements across all files by pressing "Y".

Boom.

An alternate way to avoid this problem is to use your own unique IDs to resources, and have your link type map to the video id⁵.

But for now, just replace your links using a project-wide replace function. Keep it simple until you get those brand deals.

Let's stop for now

This could go on and on. There a lot of neat workflow tricks, things you can do with RSS, CI/CD, org-publish, defining an org export mode, etc.. See other articles on this blog for more! I'm sure someone has done something using the embark package that I haven't had time to test yet.

Conclusion

Org mode provides the speed-hungry with the most drugged up concoction of stimulants to bring your emacs fueled creator journey to the next, next, next, level.

At any point you could have stopped. Any of those solutions work to a quick workflow of inserting a simple link would work. It just depends on what you're looking for, and how you want to design your workflow.

I'd also love to know if you have any tips of your own, or anything you'd like to share. I'm also at this time looking for work in software development, or just being paid to eat chips. Please get in touch at jwow@17070415.xyz.

Lastly, though I chose the path of org links, reach for macros when needed. If you need higher degrees of parameterization (you have more varying settings; for example at times wanting to use ogg format, but falling back to mp3, but sometimes there's no ogg or mp3, you can do that in the <audio> tag natively in html with more than one <source> tag. Your macro can take multiple parameters, where as in a link there isn't an obvious interface to do that).

Thanks for reading!

Footnotes:

Hyper-optimization, or self-improvement, or rampant note-taking types will also fit in.

The manual notes the most important variable is dnd-protocol-alist, a list of (protocol . action) I think a great package to see how to customize drag and drop is org-download.

You can check buffer local variables in an org buffer. I have org--dnd-multi-local-file-handler, defined by org.el. This function eventually calls this function to give that menu to attach, insert, or insert file link.

(defun org--dnd-local-file-handler (url action &optional separator)
  "Handle file URL as per ACTION.
SEPARATOR is the string to insert after each link.  It may be nil
in which case, space is inserted."
  (unless separator
    (setq separator " "))
  (let ((method (if (eq org-yank-dnd-method 'ask)
                    (org--dnd-rmc
                     "What to do with file?"
                     '((?a "attach" attach)
                       (?o "open" open)
                       (?f "insert file: link" file-link)))
                  org-yank-dnd-method)))
    (pcase method
      (`attach (org--dnd-attach-file url action separator))
      (`open (dnd-open-local-file url action))
      (`file-link
       (let ((filename (dnd-get-local-file-name url)))
         (insert (org-link-make-string (concat "file:" filename)) separator))))))

Which should mean, you could actually make a new menu item.

If your function can't handle the request of what's being dropped, you can call org-download-dnd-fallback like org-download does (link).

See Completion. This is the generic completion in org mode which you can do in front of other elements too.

⁴

(or M-TAB? I don't know anyone who doesn't have that as actually Alt-TAB~/~Cmd-TAB for switching desktop windows) The point is, fire a command to trigger auto-complete. M-x pcomplete should work. But make sure you have org-link-abbrev-alist with some candidates or there's nothing to complete.

⁵

Then in the export function, look up some mapping for ID to video link/token string. That way when video tokens don't work, you just change the ID-to-video-link association in one place. Re-export and you're good to go!.

Changing Org HTML Export, plain-list

4024-12-19T11:08:00Z

Have you ever wished your plain-lists in org mode felt more visually similar to the html rendering? I did. In one small spot.

I'll walk through my debug process, or you can skip straight to Results.

A few days ago, I went inside org html export code in ox-html.el and fixed something that had long bothered me: Inconsistent visual line breaks¹.

This is better demonstrated with a sample.

This:

- line item head 1
  this line will be conjoined in html with "line item head 1"

- line item head 2

  this line forces "line item head 2" to be wrapped in a <p>. This line will also be wrapped in a <p>.

Renders this:

<ul class="org-ul">
<li>line item head 1
this line will be conjoined in html with "line item head 1"</li>

<li><p>
line item head 2
</p>

<p>
this line forces "line item head 2" to be wrapped in a &lt;p&gt;. This line will also be wrapped in a &lt;p&gt;.
</p></li>
</ul>

Then the browser renders this:

The picture is from a basic org file without CSS. Of course on this blog I do have CSS, so the extra <p> causes more unexpectedness in such cases.

I wanted instead:

A <br> for the "\n" line item head 1, so I visually saw a separate line ("\n" isn't rendered in HTML)².
<p> s to not wrap the line item head, pretty much ever. Maybe I'm a rube for wanting that. I can solve some visual issues with CSS targetting, but semantically this makes less sense to me.

I spent one long night debugging this.

tl;dr, the "fix"

I copied org-html-paragraph from ox-html.el, and used my own export backend³ that I use with this blog.

Copy org-html-paragraph to jwow/org-html-paragraph and modify code⁴. I changed one branch of the cond.

(and (eq parent-type 'item)
                 ;; Not sure if this previous element condition is needed; might be extraneous
     (not (org-export-get-previous-element paragraph info))
                 (if (string-match-p "\n" contents) 
                                 (string-replace "\n" "<br>" contents)
                         contents))

Update the function used for paragraph transformation to jwow/org-html-paragraph in my derived backend.

;; a keyword argument to org-export-define-derived-backend
;; when a paragraph in the org ast is found, we call our function
:translate-alist '((paragraph . jwow/org-html-paragraph))

This fix is more of an opinion on what should happen. The original html export code was very intententional about when <p> insertion behavior happened, but no comments explained why.

Update 4024-12-23:

I found that I didn't like every line getting a new line. Here's what I mean:

1. plain-list para
   new line 1
   new line 2

became

<li>plain-list para<br>new line 1
<br>new line 2</li>

This visually makes new line 1 appear separate from new line 2. I realized I was used to org's handling of paragraphs where you could write text on a line immediately following the last, and it would be joined into one paragraph.

So changed the regex replacement to this (this matches the first \n found, and only it gets replaced; other \n are untouched)⁵:

(replace-regexp-in-string "\\(\n\\)\\(.\\|\n\\)*\\'" "<br>" contents nil nil 1)

Now the same org snippet gives

<li>plain-list para<br>new line 1
new line 2</li>

Now the subsequent lines are just how a paragraph would render when one has one line after another.

Debugging Process

Find out where to snag the debugger

I knew I was exporting to html, and ox-html.el was responsible.

I found the line

(plain-list . org-html-plain-list)

So that should be it I thought, try and snag that.

I ran edebug-defun on the transcoder. Then I highlighted a region and ran org-export-region-to-html over line item head 2.

When the breakpoint snagged, I pressed e and ran

(substring-no-properties contents)

which showed

"<li><p>\nline item head 2\n</p>\n\n<p>\nthis line forces \"line item head 2\" to be wrapped in a &lt;p&gt;. This line will also be wrapped in a &lt;p&gt;.</p></li>\n"

At this point I was confused–it was already in HTML by the time I got here.

The stacktrace showed calls like this:

(plist-put attributes :class (org-
(org-html--make-attribute-string (
(format "<%s %s>\n%s</%s>" type (o
(let* ((type (let* ((val (org-elem
org-html-plain-list((plain-list (:
org-export-data((plain-list (:stan
#f(compiled-function (element) #<b
mapconcat(#f(compiled-function (el
org-export-data((section (:standar
#f(compiled-function (element) #<b
mapconcat(#f(compiled-function (el
org-export-data((org-data (:standa
org-export-as(html nil nil t nil) 
org-export-string-as(#("- line ite
org-export-replace-region-by(html)
org-export-region-to-html()

It was the wrong place

Confused, I wondered what else was responsible for rendering a plain-list. A search in the same file showed hits in org-html-paragraph.

So I instrumented org-html-paragraph

With my breakpoint on org-html-paragraph, it ran before org-html-plain-list.

I also printed out the (substring-no-properties contents):

"line item head 2\n"

It's not in html yet. This must be it!

But what? Where was the paragraph in "- line item head 2"? It looked like a bullet heading to me. What was the relationship between the plain list and a paragraph?

The AST

To answer that question, we need to see what org parsed as the abstract syntax tree.

Each of the transcoders gets as input, an AST node as the first positional argument. That's what I wanted to see.

I didn't find any built in tools to dump it into a format to see the relationships between elements⁶.

It was only until I found the screenshot on the readme of org_mode_ast_investigation_tool that I realized a plainlist contained an item, and an item contained a paragraph, which was the level in the tree I actually cared about.

Making changes

The rest is in tl;dr, the "fix", where I did some string manipulation..

Results

This:

- line item head 1
  this line will be conjoined in html with "line item head 1"

- line item head 2

  this line forces "line item head 2" to be wrapped in a <p>. This line will also be wrapped in a <p>.

Renders this:

<ul class="org-ul">
<li>line item head 1<br/>this line will be conjoined in html with "line item head 1"<br/></li>

<li>line item head 2<br/>

<p>
this line forces "line item head 2" to be wrapped in a &lt;p&gt;. This line will also be wrapped in a &lt;p&gt;.
</p></li>
</ul>

Then the browser renders this:

🚀⁷

Final Thoughts

It would be great if the AST in org mode were easier to visualize within emacs, using its own built in tools. I couldn't make sense of it with the giant print out easily.

I'm not the best with introspecting objects, so I got stuck on what was considered a "paragraph."

Is there also no stacktrace ability to jump to source of caller? This can be a little difficult with macros, but for standard functions this should be possible. I couldn't find this ability anywhere in edebug's docs. Discord also was crickets.

I also really like the manual, and I think certain sections deserve an update. Export mostly just works.

Footnotes:

I didn't want the behavior of org-export-preserve-breaks.

To visually render a newline, you need a line break element: <br>. It looks like org mode intentionally kept the "\n". I say this because other elements like paragraphs in org actually remove the new lines. I like it because this is good for source control. (Try it!)

See org vocabulary for definitions.

⁴

(defun jwow/org-html-paragraph (paragraph contents info)
  "Transcode a PARAGRAPH element from Org to HTML.
    CONTENTS is the contents of the paragraph, as a string.  INFO is
    the plist used as a communication channel.
    Same as org-html-paragraph, but list items render 
    "
  (let* ((parent (org-element-parent paragraph))
         (parent-type (org-element-type parent))
         (style '((footnote-definition " class=\"footpara\"")
                  (org-data " class=\"footpara\"")))
         (attributes (org-html--make-attribute-string
                      (org-export-read-attribute :attr_html paragraph)))
         (extra (or (cadr (assq parent-type style)) "")))
    (cond
     (
      ;; This condition is when the paragraph is actually the text of the line item in the AST
      ;; Then we simply return contents but replace newlines with <br>
      (and (eq parent-type 'item)
                                         ;; Not sure if this condition is needed; might be extraneous
           (not (org-export-get-previous-element paragraph info))
           ;; (let ((followers (org-export-get-next-element paragraph info 2)))
           ;;   (and (not (cdr followers))
           ;;        (org-element-type-p (car followers) '(nil plain-list paragraph))))
                                 )
      ;; First paragraph in an item has no tag if it is alone or
      ;; followed, at most, by a sub-list.
                (if (string-match-p "\n" contents) 
                                (string-replace "\n" "<br>" contents)
                        contents))
     ((org-html-standalone-image-p paragraph info)
      ;; Standalone image.
      (let ((caption
             (let ((raw (org-export-data
                         (org-export-get-caption paragraph) info))
                   (org-html-standalone-image-predicate
                    #'org-html--has-caption-p))
               (if (not (org-string-nw-p raw)) raw
                 (concat "<span class=\"figure-number\">"
                         (format (org-html--translate "Figure %d:" info)
                                 (org-export-get-ordinal
                                  (org-element-map paragraph 'link
                                    #'identity info t)
                                  info nil #'org-html-standalone-image-p))
                         " </span>"
                         raw))))
            (label (org-html--reference paragraph info)))
        (org-html--wrap-image contents info caption label)))
     ;; Regular paragraph.
     (t (format "<p%s%s>\n%s</p>"
                (if (org-string-nw-p attributes)
                    (concat " " attributes) "")
                extra contents)))))

⁵

Nothing humbling like regexp. I totally forgot "." wouldn't match on a literal "\n" in the string, which is why it's ored with a newline. The \\' in the regexp is special to (replace-regexp-in-string), as it allows you to replace only the first match. Check the help. C-h v replace-regexp-in-string.

⁶

Edebug and the backtrace can help some.

When in edebug, I pressed d, which pops us into the *Edebug Backtrace* buffer.

The stacktrace was not very readable for this snippet:

org-html-paragraph((paragraph (:standard-properties [3 3 3 20 21 1 nil nil nil nil nil nil nil nil #<buffer  *temp*-992987-822380> nil nil ...])

Pressing + calls backtrace-multi-line at the start of an expression can expand properties. Pressing Enter on ellipses can show

org-html-paragraph((paragraph
                    (:standard-properties
                     [3 3 3 20 21 1 nil nil nil nil nil nil nil nil #<buffer  *temp*-984820-519693> nil nil
                        (item
                         (:standard-properties
                          [1 1 3 123 123 0 (:tag) item nil nil nil nil nil nil #<buffer  *temp*-984820-519693> nil
                             ((1 0 "- " nil nil nil 123))
                             (plain-list
                              (:standard-properties
                               [1 1 1 123 123 0 nil top-comment nil nil nil nil nil nil #<buffer
                                  *temp*-984820-519693> nil (...) (section ... #11)]
                               :type unordered)
                              #6)]
                          :bullet "- " :checkbox nil :counter nil :pre-blank 0 :tag nil)
                         (paragraph (:standard-properties #5) #("line item head 2\n" 0 17 (:parent #9)))
                         (paragraph
                          (:standard-properties
                           [21 21 21 123 123 0 nil nil nil nil nil nil nil nil #<buffer  *temp*-984820-519693> nil nil
                               #6])
                          #("  this line forces \"line item head 2\" to be wrapped in a <p>. This line will also be wrapped in a <p>."
                            0 102 (:parent #10))))])
                    #("line item head 2\n" 0 17 (:parent (paragraph (:standard-properties [3 3 3 20 21 1 nil nil nil nil nil nil nil nil #<buffer  *temp*-984820-519693> nil nil (item (:standard-properties [1 1 3 123 123 0 (:tag) item nil nil nil nil nil nil #<buffer  *temp*-984820-519693> nil (...) (plain-list ... #12)] :bullet "- " :checkbox nil :counter nil :pre-blank 0 :tag nil) #7 (paragraph (:standard-properties [21 21 21 123 123 0 nil nil nil nil nil nil nil nil #<buffer  *temp*-984820-519693> nil nil #12]) #("  this line forces \"line item head 2\" to be wrapped in a <p>. This line will also be wrapped in a <p>." 0 102 (:parent #16))))]) #4))))

This gave clues, but wasn't easy to comprehend. The :standard-properties are hard to parse out, as they're in a vector and I had no idea what each position meant.

⁷

If you're paying super close attention, you'll notice that in the org, theres a newline after "this line will be conjoined…" but no visual new line in the browser render. That's ok for me. Most newlines in an org doc don't warrant a visual new line. I often have paragraphs of comments to myself on how to reword a section, or scratch notes, which are spaced with new lines.

If I need to force a newline just for rendering, I'd type "\\ " at the end of my line in org mode which will convert to a <br>.

A Transient Matter

4024-12-18T05:20:00Z

2000 years ago, I wrote this for my present self:

It's 5:20 AM and I have been debugging for what has been 2 or maybe some odd hours while tired. My sleep schedule is no more. Seriously, why didn't you reach out with a hand that can traverse temporal dimensions and stop me from doing this? All for ergonomic keybindings. The calculus is unjustified. I'm going to bed and I'll finish this post later.

This is obscure. It applies to users of xah-fly-keys and actions on a transient.

Jump to tl;dr if you'd like

What's the problem I couldn't tear away from aeons ago? The transient UI in rg.el–I've had this installed for probably close to a decade yet I never realized it came with a transient UI. In this UI, you can set flags. An example: typing - m lets you change the --max-count flag to only get that many matches per file. You're dandily presented a minibuffer with --max-count= as a prompt. You type in 2, because you only want two matches per file.

I pressed <Enter>. And this is where suffering begins…

Look behind you! How did you get here? What walls – wha? You were out simply noodling with your fishing rod off by the village pond, with your evil vim-loser friend friend trying to tell you about how great Rust is, and how you should rewrite all your programs in Rust!

But these walls. How long have they been there, these mossy, dust covered slabs? A summon spell must have brought you here. Well, into the dark maze you go then. As you step into sure abyss, you lean one hand on the wall for stability. Wait. Runes? You touch the stonework for a closer look. A magic panel lights up! Oooh–quickly! Touch all the buttons!

You hear what must be fated rumbling. An inevitability you've activated. Is it the low hum of menace? Is it a growl of a bear? Mayhap a god or spirit or troglodyte you've angered. You try to turn on the light to see clearly. In the middle of twiddling with a wet match, you hear a squeak escape your throat.

"Oh. Fs" You hear demon song:

Unbound suffix: ‘g’ (Use ‘C-g’ to abort, ‘?’ for help) [set-mark-command] [5 times]

Why isn't it working? The runes–there–'g' means go. I'm 'g'ing!

The bass section of the demon chant grows louder. Somewhere in the labyrinth you think you hear an undead angel soprano singing the sweeping melody.

Unbound suffix: ‘r’ (Use ‘C-g’ to abort, ‘?’ for help) [open-line]
Unbound suffix: ‘t’ (Use ‘C-g’ to abort, ‘?’ for help) [xah-fly-insert-mode-activate]
Unbound suffix: ‘f’ (Use ‘C-g’ to abort, ‘?’ for help) [backward-kill-word]
Unbound suffix: ‘d’ (Use ‘C-g’ to abort, ‘?’ for help) [xah-delete-current-text-block]
Unbound suffix: ‘b’ (Use ‘C-g’ to abort, ‘?’ for help) [xah-toggle-letter-case]
Unbound suffix: ‘w’ (Use ‘C-g’ to abort, ‘?’ for help) [xah-shrink-whitespaces]

Your jaw clenches. The adrenaline jolts your heart like a flat punch. For the first moment in your life you wonder if it would matter if you died, if anyone besides your pet goldfish would ever care.

You've bought all these democracies, set other countries to war for profit, destroyed the healthcare system for trillions, and this is this how you die?

No time to think–once more with feeling!

Unbound suffix: ‘e’ (Use ‘C-g’ to abort, ‘?’ for help) [wgrep-change-to-wgrep-mode]

With your mistap wgrep squirms out of your pocket. You shove it back in, thankful the safety mechanism prevented it from discharging over your buffers. It had cost your grandmother's life to tame that beast, trap it in a jar, and distill its vengeful spirit in a homunculus fashionably bound to your 'e'. Like a piece of bijoux.

Keep calm! Keep cool!

What about C-h k?

Unbound suffix: ‘k’ (Use ‘C-g’ to abort, ‘?’ for help) [isearch-forward]

k? Why had it been orphaned from C-h?

Keep calm! Keep cool!

You'll C-h m and see what–.

You're brought to a man page for rg. No! Quit! How'd you get here, manpage?

You successfully quit the man page, but why was your trusty C-h m not working? Was it caused by a flipped bit from gamma radiation? You try again.

Oh no! It won't stop!

Your beloved C-h m (describe-mode)! Not even help can be summoned!

You thrash your fingers in counter rhythm to the doomful dirge. Something, you have a feeling, wants your bones and organs. Wants them more badly than you do. A stroke of panic invades every cell in your body. Screw what the rg runes say!

Unbound suffix: ‘e’ (Use ‘C-g’ to abort, ‘?’ for help) [wgrep-change-to-wgrep-mode]
Unbound suffix: ‘n’ (Use ‘C-g’ to abort, ‘?’ for help) [next-error-no-select]
Unbound suffix: ‘i’ (Use ‘C-g’ to abort, ‘?’ for help) [rg-rerun-toggle-ignore]
Unbound suffix: ‘u’ (Use ‘C-g’ to abort, ‘?’ for help) [self-insert-command]
Unbound suffix: ‘e’ (Use ‘C-g’ to abort, ‘?’ for help) [wgrep-change-to-wgrep-mode]
Unbound suffix: ‘i’ (Use ‘C-g’ to abort, ‘?’ for help) [rg-rerun-toggle-ignore]
Unbound suffix: ‘n’ (Use ‘C-g’ to abort, ‘?’ for help) [next-error-no-select]

But your legs are pinned. The inverted T navigation keys prove no use. A crimson light floods the hall, dialates your pupils into saucers, blinds you. A dark shadow clad in glowing rune armor begins sprinting at you from about a hundred meters away. You know only from sound. The rumbling deepens; you feel the hollowness of your chest vibrating. The floor quakes from each footstep and dust from the ceiling irritates your eyes. Your whole body tightens and your forearms flex like steel cables as you pull out your enchanted nunchucks. You don't know how dicey this is going to get, but there is no way, not through all the recursion-depth-limit-exceeded hell you've been through, that you're going to press abort.

Your trump card!

Unbound suffix: ‘<esc>’ (Use ‘C-g’ to abort, ‘?’ for help) [xah-fly-command-mode-activate] [3 times]

You hear the sinister voice mock you.

You cannot escape from me! Your spell chest is mine! Mwahahaha...! Without your spells you would be a spec of dust, a
life sentenced to the drudgery of working paycheck to paycheck! Soon your people will be devoured, they will--

Blast! But before you have time to scrounge into your pocket, the demon has zipped a spear through space-time through your abdomen. Oddly you don't feel physical pain.

You look down and see the dark knight lifting your impaled body at the hilt. Your senses start to warp; you taste pain, smell light, hear death.

"Hey! Are you still there? Don't die on me," you hear from me, the narrator.

Your eyes start to float gently away from your skull as you see your hand feebly itching for one last motion.

You think to cast the forbidden emergency town portal: C-g. Is it pride, luck, or complete discombobulation? Your ring finger slips and C-z activates. The shadow is gone with nothing but quiet remaining. You find yourself panting, unsure if your heart was going to explode. You take a moment to read through *Messages* as you bleed out. It's cold, but it'll be over soon.

"Take the rest of me, Death" you declare. You check C-h l for your spell history, which seems to work. Ahah, there!

C-z            ;; transient-suspend

"Suspend? So that which has stripped me of all my glory yet still lives." You lose consciousness. Softly as you fade, you become aware of a jolly tune by Mariah Carrey.

You skip the scene where death delays your passing because of a bureaucratic Christmas technicality. Unfortunately the Death judiciary has slowed dramatically to curb "illegal immigration." You may not seek asylum, and awaiting trial will at least a few years. You skip this story until this paragraph:

You finally escape the labyrinthian world of clashing mode maps, where the swerve of emacs physics was fast melting your hands and eyes. Catch your breath. Recompose. Echoes of your dead spells haunt your mind. You are only a few of the survivors of the Ritual of Unbound Suffix, nemesis of all holy and just.

How would you, dear reader, belay this labyrinth, and bring balance to Emacs Land?

tl;dr

For those who use xah-fly-keys (there's less than 200 of you on planet Earth), and you struggle with transients messing with your keybindings, I've found a temporary solution to get around the issue of the keys being snagged.

;; UPDATED 
;; 2024-12-25, I found some bugs that I fixed

;; If you want to add this code to your config, you want to make sure
;; xfk is loaded FIRST, because we need to remove a hook that it adds
;; (xah-fly-keys 1)

;; xfk adds this hook in library, you want to remove it since it triggers command mode on ANY minibuffer exit
;; and when it happens in transients, you get stuck. You want to be in insert mode still
(remove-hook 'minibuffer-exit-hook 'xah-fly-command-mode-activate)
(add-hook 'minibuffer-exit-hook #'jwow/transient-aware-activate-xfk)
;; This variable transien--showp actually is truthy when transient is activet.
;; So if it is active, we don't want to switch out of insert mode
;; In all other cases, this is just activating command hook on minibuffer exit
(defun jwow/transient-aware-activate-xfk ()
  (when (not transient--showp)
    (xah-fly-command-mode-activate)))

;; The transient-exti/setup hooks are are optional, 
;; they don't always do the most convenient thing
;; Bring yourself back to command mode once ALL transients are popped out of
;; don't use xah-fly-command-mode-activate, as leaving a nested transient will exit
;; into command mode when you're in the parent transient
(add-hook 'transient-exit-hook #'jwow/transient-aware-activate-xfk)
;; Always switch to insert when activating a transient
(add-hook 'transient-setup-buffer-hook #'xah-fly-insert-mode-activate)

;; git-commit from magit for example, will put you into command mode when you're in the
;; git commit buffer, you can use this if desired:
;; (add-hook 'git-commit-mode-hook #'xah-fly-insert-mode-activate)

transient--showp is a private variable, but it comes with a doc string so this should be relatively safe to use externally for now.

Whether to show the transient popup buffer.

Does it enable popup buffers? When is it truthy? I wasn't sure if it was it, so I had to just try it.

How did I verify that this was the variable to use among all of transient variables? I instrumented jwow/transient-aware-activate-xfk with edebug, used my workflow for rg, the breakpoint snagged, and this let me look for all variables matching ^transient- and check their state during the transient's activation. After exiting the transient, I then refreshed each help buffer for each variable. Yep, mildly painful.

Anyway that's all. Frankly I'm surprised I solved this issue. Time to sleep, and hopefully to dream, free from the Curse of the Unbound Suffix :D.

In the shadows, magit-transient promises to avenge its fallen kin…

UPDATE 2024-12-25: Changed some code. I ran into a nested transient issue (try magit, ?, M), which brings up the remote transient, after the help transient. I got stuck in command mode but needed to be in insert mode.

Fast Workaround for Conflicting Project Dependency Loading In Emacs

4024-12-16T22:20:00Z

This error message is new to the bleeding edge of emacs, but the error itself, the loading of dependencies of the same name, is not:

condition-case: Eager macro-expansion failure: (error "Feature ‘project’ loaded from \"/usr/share/emacs/31.0.50/lisp/progmodes/project.elc\" is now provided by \"~/.emacs.d/straight/build/project/project.elc\"")

I had this error in dealing with eglot and dape loading in certain packages, one from straight, and one that's built in.

tl;dr

Each time I have a conflicted load, I've done this before loading in any other package in my init.el:

(use-package project
  :straight (:type built-in))
(use-package xref
  :straight (:type built-in))
(use-package jsonrpc
  :straight (:type built-in))

Working ok so far. Of course, I'm on bleeding edge emacs, which has jsonrpc 1.0.25, so dape can use it. If you're on emacs 29.4, the built in is still 1.0.24. In which case, you might want to specify to use your most up to date version of jsonrpc:

(use-package jsonrpc
  :straight 't)

Describe Mode Outline

4024-12-14T01:03:00Z

I learn keys using C-h m, bound to describe-mode. And I also check what modes I'm in if I certain keybindings don't seem to be working. It's this latter case that in an edge build of emacs (30.0.50), something has changed.

Before, describe-mode would helpfully list all active modes up front–easy to scan.

Minor modes enabled in this buffer: Auto-Save Centered-Cursor
Company-Box Company Font-Lock Org-Bullets Org-Indent Visual-Line Yas

The major mode is Org mode defined in org.el:

Outline-based notes management and organizer, alias
"Carsten’s outline-mode for keeping track of everything."

But now this text that shows all modes is gone. I'm not sure if it's because I upgraded emacs 30 or something else, but describe mode now has an outline. I found the variable by autocompleting for describe- and found describe-mode-outline, which you can customize and turn it off. Phew.

Docker Container Volume Not Writable

4024-12-14T00:15:00Z

Say you clone a repo, you run docker-compose up, it pulls down packages, and you finally visit the app on localhost.

And then disaster strikes–500 errors. It turns out some directory isn't writable from the app. In my case, codeigniter needed to write a cache file to the mounted volume.

tl;dr

tl;dr the remedy I used was to

Shell into the container: docker exec <container-name> -it sh
Once you're inside the container like this, I was root in the subvolume. You can run the rest of the commands from the container. Of course, you can do it outside of the container too.

(Note this doesn't always work because not all containers will have a shell–but they usually do.)
Find the user running the process in question (read thru the docker entrypoint file of the container)
```
# this lists the user who owns the process on the left-most column
ps aux | grep <program-name>
```
If ps isn't installed, this might not work, in which case you can read docs on what the user would be by looking through the dockerfile for the image of the container. You can also cat /etc/passwd if that helps you narrow things down.
Find what group the user was in, and set the group for all files recursively
```
id <user-name>
```
Should return something like uid=33(http) gid=33(http) groups=33(http)

Change the group for your repository

chgrp -R <group-name> /path/to/subvolume

Set the rwx group permissions
```
chmod g+rwx /path/to/subvolume
```

And then I had no errors!

Interestingly I had a user called http since I run apache2 (also called httpd) outside container. And in the container had a user called www-data. When I ran ls -la from my host machine, I saw the group was http. When I ran the same command inside the container, I saw the group was www-data, which was interesting.

Anyway, hope this helped.

(FWIW I don't remember having this problem on mac, so this might be something you tend to run into while in linux.)

Emacs Line length annoyances

4024-12-11T04:55:00Z

Emacs truncates messages when I'm debugging elisp. It has always annoyed me. I never found the setting to not truncate because I always included "message" as part of the string, thinking it was an echo area / messages buffer setting.

The default emacs line length variables should just be higher by default.

eval-expression-print-length defaults at 12! That's borderline unusable when dealing with strings and file paths.
edebug-print-length defaults to 50.

Setting those to at least 120 offers me some sanity back.

I'm glad I found this post on emacs.stackexchange.com. Maybe I should use apropos more.

Project-specific On-Save Hooks in Emacs

4024-12-10T00:00:00Z

Imagine if you wanted some process to run automatically, tailored to your project, after you saved your changes. For me it's exporting an org-mode file to a file 11ty can render. How is that done in emacs?

tl;dr

Get a .dir-locals.el file that holds dotted pairs like this:

(org-mode . ((eval . (add-hook 'after-save-hook #'org-11ty-export-to-11tydata-and-html nil t))))

Explanation

Dir local variables are inherited by nested directories. So at the top level of a project, all files and files in directories under the .dir-locals.el will have trigger my command on save.

This means that in all my org mode buffers, exporting the org file happens automatically after I save. In my case, I generate files that 11ty uses. (As for why I'm using 11ty AND org-mode to make the site and not just one or the other, it's technical debt).

These variables of course, don't have to introduce runnable code (via the eval). They can just hold state:

;; This just sets a variable org-confirm-babel-evaluate to nil for all modes
(nil . ((org-confirm-babel-evaluate . nil)))

Read more in the manual: emacs#Directory Variables.

Ergonomics

add-dir-local-variable: You can interactively add dir-local variables instead of manually editing the .dir-locals.el.
copy-file-locals-to-dir-locals: When you specify file specific variables, you can just have emacs put them into .dir-locals.el for you.
Example:

I often have local file variables like below, under a commented header so it doesn't export. I can run the command to have this variable put into .dir-locals.el for me:
```
* COMMENT Local Variables
;; Local Variables:
;; flycheck-disabled-checkers: (proselint vale)
;; End:
```

Endless parenthesis covered this topic too.
Use .dir-locals-2.el if .dir-locals.el is committed to a repository and you don't like dangling changes. Found this tidbit in the manual.

PHP in Leetcode

4024-12-06T00:00:00Z

Today marks my 300th day streak in leetcode. I did today's problem in PHP because I want to volunteer for nonprofits. A lot of nonprofits use PHP (and frameworks on top of PHP like wordpress).

This post about my experience using PHP in leetcode, not actually the algorithms.

Anyway, how is PHP on leetcode? It's awful. I find that when I make submissions I beat 100% of all users because no one else submits in PHP, and I realized why.

Let's just look at some code for the daily on 2024-12-06: 1760. Minimum Limit of Balls in a Bag

The Problem

Cutting to the chase, binary searching over the solution space is the recommended approach. I wasn't able to come up with a heap solution, but if I did, I would reach for a new SplMaxHeap(). The standard library is thankfully loaded, so no need to fuss with importing.

Below I assume you're familiar with this technique. If not, it's basically 2 things:

A validation function to check if the guess is correct
A binary search that uses the validation function

The invariant used for the problem is monotonicity (monotonicicity isn't always the invariant used in binary search, but probably the most common. Just check out the find-peak-element problem as a canonical example of an invariant that isn't monotonicity).

Code

Setting up the validation function

class Solution {

    /**
     * @param Integer[] $nums
     * @param Integer $maxOperations
     * @return Integer
     */
    function minimumSize($nums, $maxOperations) {
        $check = function (int $allowedSize) use (&$nums, $maxOperations) {
            $ops = 0;
            foreach($nums as $num) {
                // echo ' adding: ', ceil($num / $allowedSize) - 1, PHP_EOL;
                $ops += ceil($num / $allowedSize) - 1;
            }
            // echo 'ops: ' . $ops . ' ' . $maxOperations . PHP_EOL;
            return $ops <= $maxOperations ;
        };

This problem reminded me of how explicit you have to be with closures. It's the use(&$nums, $maxOperations) that allows the anonymous function to reference those variables from the outer lexical scope. Coming from ruby, python, javascript, lisps, (even java lambdas), this just seems really unecessary.

What's also horrible on the leetcode platform is that if you don't have a variable defined, no exception will throw. It took me a few minutes to realize from echo ing that in fact, $nums and $maxOperations were both nullish (if that's the correct term in php–they don't print anything). I had to use the use syntax.

I also ran into this newbie gotcha–I couldn't declare an inner function like this:

function check (int $allowedSize)

because on running other test cases, the interpreter errored out saying check was already defined. Yes, functions have global scope, don't expect lexical scoping to kick in here.

Writing a binary search

I follow a standard binary search template. I like this one the most because it's symmetric unlike other templates. I use $check as my validation function to see if the current guess works, and move $hi or $lo accordingly.

        $lo = 1;
        $hi = 1e9;
        $best = $hi;
        while ($lo <= $hi) {
            $mid = $lo + floor(($hi - $lo) / 2);
            // echo 'mid: ' . $mid . PHP_EOL;
            if ($check($mid)) {
                $hi = $mid - 1;
                $best = $mid;
            } else {
                $lo = $mid + 1;
            }
        }
        return $best;
    }
}

Issues

It's not ergonomic to echo PHP_EOL when you want to debug via printing. People seem really hesitant to add some kind of println utility to the language. Sigh.
PHP provides no built-in binary search :(. I wouldn't have used it anyway for this problem though since we're not binary searching over an array, but an abstract search space.
Leetcode also doesn't add any useful libraries that would help. The only module included is bcmath, which…is not helpful in the slightest for almost all problems on the platform. Sometimes you just want an ordered map¹ (balanced binary trees implement this).
PHP feels more like a templating language still, rather than an application server language, making it not a preferred fit for leetcode.
PHP has a lot of surprises in its own functions
- Like argument order of very related functions
  - array_map takes the callback first
```
function array_map(?callable $callback, array $array, array $arrays = unknown): array
```
    and ugh, ok ok, its variadic to how many arrays you pass in so the callable comes first, but this was totally unexpected.
  - array_reduce takes the array first
```
function array_reduce(array $array, callable $callback, mixed $initial = null): ?mixed
```
- the naming of functions themselves: strlen vs str_pad, how do you reason about when underscores happen?.
- You cannot have a callable as a type hint for an instance variable….
- Passing functions around seems like an afterthought of the language. Maybe I'll get used to it? You either pass in a string, which references the name of a function, an anonymous function declared directly, or a formatted array (format varies depending on what you're calling).

Conclusion

I wrote a draft of this post days ago. I think I'm beginning to kind of love php in a weird way. The tooling is there in emacs. Maybe I'll post about that later.

Leetcode is a great platform to practice language syntax. I've reviewed quite a few concepts just doing a few problems.

You can try yourself, and reach for these datastructures on leetcode without importing anything:

SplDoublyLinkedList  SplHeap              SplObserver          SplSubject
SplFileInfo          SplMaxHeap           SplPriorityQueue     SplTempFileObject
SplFileObject        SplMinHeap           SplQueue             
SplFixedArray        SplObjectStorage     SplStack

I also learned a lot by packaging a php binary index tree (it's pretty alpha, I'm still learning). One could argue one would not need a library for such a small data structure. But I made it anyway–no one else did for php, might as well. Also on packagist, which is the popular(?) package repository for php.

Extra Problems

This section is for a cursory glance a few different uses of the Spl datastructures on leetcode, without explanation of the algorithms. PHP is pretty capable, but I'll stick with python usually.

2558. Take Gifts From the Richest Pile

In this problem, you want a container with orderedness as you mutate it. Trees work (no builtin available). But heaps are faster.

PHP's built in heaps don't have a heapify method like in python to turn an array into a heap in O(n).

You also don't have something like heappushpop or heapreplace, which are faster than a single heappop.

class Solution {

    /**
     * @param Integer[] $gifts
     * @param Integer $k
     * @return Integer
     */
    function pickGifts($gifts, $k) {
        $heap = new SplMaxHeap();
        foreach($gifts as $gift) {
            $heap->insert($gift);
        }
        while($k > 0 and $heap->top() > 1) {
            $max = $heap->extract();
            $insert = (int)sqrt($max);
            // echo $max, ' ', $insert, PHP_EOL;
            $heap->insert($insert);
            $k--;
        }
        $res = 0;
        foreach($heap as $gift) {
            $res += $gift;
        }
        return $res;
    }
}

2577. Minimum Time to Visit a Cell In a Grid

And you can of course do dijkstra with the SplMinHeap. You can also solve this using Dial's algorithm, a dijkstra variant, by using SplQueue or even SplDoublyLinkedList if you want to go that route.

Here my heap elements are (curr_cost, row, column). This shows you that the heap behaves like heaps in python. Elements will be compared index-wise. That's why I didn't need a custom comparator–just store more data in a heap node.

class Solution {
    /**
     * @param Integer[][] $grid
     * @return Integer
     */
    function minimumTime($grid) {
        $m = count($grid); $n = count($grid[0]);
        $able_to_visit = 0;
        if ($m != 1) {
            $able_to_visit += $grid[1][0] <= 1 ? 1 : 0;
        }
        if ($n != 1) {
            $able_to_visit += $grid[0][1] <= 1 ? 1 : 0;
        }
        if ($able_to_visit == 0){
            return ($n == 1 and $n == 1) ? 0 : -1;
        }
        $min_costs = array_fill(0, $m, array_fill(0, $n, PHP_INT_MAX));
        $min_costs[0][0] = 0;
        // var_dump($min_costs);
        $q = new \SplMinHeap();
        $q->insert([$grid[0][0], 0, 0]);
        while ($q) {
            [$curr_cost, $r, $c] = $q->extract();
            if ($r == $m - 1 and $c == $n - 1) {
                return $curr_cost;
            }
            foreach ([[$r, $c + 1], [$r, $c - 1], [$r + 1, $c], [$r - 1, $c]] as [$next_r, $next_c]) {
                if ($next_r < 0 or $next_r == $m or $next_c < 0 or $next_c == $n) {
                    continue;
                }
                $next_cost = $grid[$next_r][$next_c];
                # take into account back,forward steps
                if ($curr_cost + 1 >= $next_cost) {
                    $next_cost = $curr_cost + 1;
                } else {
                    $next_cost += ($next_cost - ($curr_cost + 1)) % 2;
                }
                if ($next_cost < $min_costs[$next_r][$next_c]) {
                    $min_costs[$next_r][$next_c] = $next_cost;
                    $q->insert([$next_cost, $next_r, $next_c]);
                }
            }
        }

        return -1;
    }
}

862. Shortest Subarray with Sum at Least K
Finally we can also SplDoublyLinkedList as a queue, stack, or in this case, a deque:

class Solution {

    /**
     * @param Integer[] $nums
     * @param Integer $k
     * @return Integer
     */
    function shortestSubarray($nums, $k) {
        $res = PHP_INT_MAX;

        $prefix = new SplDoublyLinkedList();
        $acc = 0;
        foreach($nums as $i => $num) {
            $acc += $num;
            if ($acc >= $k) {
                $res = min($res, $i + 1);
            }
            // We now check if we have a window where we made a sum >= k
            // if so, try trimming the deque from the left end until we're the window sum
            // is < k. Each step, we can record the window length.
            while ($prefix->count() and $acc - $prefix[0][0] >= $k) {
                $res = min($res, $i - $prefix[0][1]);
                // remove from left end
                $prefix->shift();
            }

            // Because of negative numbers, the running sum, acc, can be less than it was
            // We want our prefix window to be monotonically increasing so we pop from right
            while ($prefix->count() and $acc <= $prefix->top()[0]) {
                // remove from right end
                $prefix->pop();
            }
            $prefix->push([$acc,$i]);
        }

        if ($res != PHP_INT_MAX) {
            return $res;
        }
        return -1;
    }
}

Footnotes:

"An array in PHP is actually an ordered map. " – from docs. However, I don't see any way to initialize one with a custom comparator to keep elements in sorted order AS THE ARRAY IS UPDATED. A java TreeMap would let you do this.

4024-10 In Review

4024-10-17T00:00:00Z

My meaningful achievements this month:

Being alive. Logistics of living continue to take a huge toll whenever I need to travel long distances.
- Taking the metro in Paris to CDG was uncomfortable/sweaty/hectic. The traincars were so crammed. One of the trains stopped and asked us to please not smoke during the delay.
- Flying was ok. I appreciated the on flight meals. I also brought a ton of boiled eggs with me.
- I didn't feel horrible for losing a lot of money on missing my connection in LAX…
I went on multi-hour bike rides to different castles, then had to bike home
- Château de Brissac
- Château de Brézé
- learned about falun (marine Cenozoic deposits) in Doué-la-Fontaine
- Went to a mushroom museum, sculpture museum, and medieval gardens
- Learned how sparkling wine was made in the region from visiting some cellars
My airbnb host this month was really nice. She wrote me a nice note after I left.
Finished a Duolingo section. Maybe 10 years ago I finished French, and now it feels like there's more content or I've just forgotten a whole bunch.
Graded some exams for a little money on the side.
I got really lost and the sun had set when I was out biking one day. I couldn't see. But I made it back.

Meaningful losses:

I haven't been writing
No opportunity creation for myself for networking. It's hard to talk to people. I'm also in the countryside.
My self-connection in LAX didn't work out. I ended up missing my flight by maybe 10 minutes. I paid for a different plane ticket I couldn't afford. Makes me feel almost regretful to stop and help someone at the airport.
Still no job/health insurance/income.
I don't feel like myself in a lot of ways.
I do not have income to feel comfortable replacing
- my 9 year old phone
- my laptop
- a lot of things that are worn out now, but that I'll continue using
It is hard to enjoy things knowing there is little to nothing on the other side of France. No place to get a job and get medical insurance in the US. It's complicated.

What I'm grateful for:

Bicycles and bike lanes
The helpful bicyclist on 2024-10-13 who stopped by the side of the road and asked if I needed any help when I was many km from familiarity
Cool places to visit are in general < €12 (castles, caves, museums)
The rain in Saumur reminds me of the rain patterns in Seattle.
La Loire. It's beautiful enough to make a cartel-tourist business offering visits to the afterlife.
The quietness of the countryside
The efficacy of oatmeal + eggs for sustained energy
Continued gratefulness for grocery prices (while finishing this post in the US, I am even more grateful. I miss it.)
Discovered that I like Bordeaux wines. I probably drank a personal record amount of alcohol this month. Almost 4 bottles in a month, which is a lot for me.
Discovered that I really really like this multi-grain bread sold by Intermarché.
Dominos pizza Thursday deals. A medium was €9 and a large was €11, and you get really good cheeses on it. Shout out to both pizza au quatre fromages and pizza au cinq fromages.
I had a chocolate stuffed beignet while biking home one day.
The french policewoman decided to not fine me or arrest me. I was recording an audio note on my phone while biking on the road. I think she thought I was talking to someone else on the phone. I'm not sure what the legal distinction for grounds of arrest would be between those two actions…
Upon leaving, my host drove me to the train station so I didn't have to do the 1hr walk with all my bags that I had planned.
My friend letting me stay for a while in the US, while I get my passport renewed

Looking ahead:

Paying a lot of money out of pocket for various medical care.
- I tried to reach out to dentists by going to their office in person and leaving my number and email. I was discouraged after a couple attempts. No one reached back out to me. I did not press–after all I do not pay taxes to France to feel like I should press.
At time of writing, I haven't returned to the United States, but I don't know what happens after that.
United States Presidential election…I was hoping I would have a new job at the same time VP Harris would. Not sure if she'll win, but I really hope so.
A continued dead job market for software engineers like myself.
I still have yet to truly create an engine (a positive feedback loop) of connection, possibility, and meaning. I do not have the resource of other people. I don't have access to all the necessary raw ingredients in the US I think.
I am and have been homeless for months now. Death does not scare me. Living does.

4024-09 In Review

4024-09-27T00:00:00Z

heyllo Wow, I didn't write anything for this month? I thought I did. I'll update this. Probably because cloudflare very much rescinded an offer of employment.

My meaningful achievements this month:

Meaningful losses:

Like last month, I've lost lots of sleep. I think I can start recovering here though.

What I'm grateful for:

Looking ahead:

Due to passport expiry, I will have to come back to the US.

4024-08 In Review

4024-08-31T00:00:00Z

My meaningful achievements this month:

Staying alive in France, and doing on average, at least an hour of Duolingo daily.
Presented, and edited a video for Dynamic Programming recurrence relations,
- which took in total, over 20 hours just for the editing and re-recording of sections where I made mistakes. I'm not going to count all the hours for this one. It was so much work!
- Making the slides, solving algorithm problems, and deciding how to categorize patterns took time that isn't counted here.
- The year is 4XXX and I still don't know how how to edit out my loud breathing, or prevent the mic from picking it up in the first place. I left my Shure mic in the Bay Area.
Finally Edited, and re-recorded my presentation on Dijkstra's Algorithm from the in-person seminar last month.
- This also took a long time, easily over 12 hours. I'm grateful for KDEnlive, because I don't have a powerful machine to edit video.
Survived living with a chainsmoker for 11 hot summer days who refused to smoke outside. I still have some heart pain, but that will go away. Breathing was really difficult at times.
- I found a cigarette butt on the kitchen floor. I have gazed upon no cigarette so demoralizing, so death-beckoning, so apathy-inspired as that one. I picked it up and threw it in the trash. I'm not sure what I did afterward, but I had so many regrets about booking this place.
- I managed my productivity these days. Some things got done, just not as much as I'd like.
- People would also smoke outside my window. I couldn't escape at times if I was indoors.
- I also cleaned out the fridge because he didn't when he left. For a 25-year-old man to live like this, it's usually because he didn't grow up with those who cared about him.
- I alerted property management, who came in on another day, and opened his door unnanounced while he was smoking. What followed was an angry scolding in French that was too fast for me to follow but definitely "Tu fumais dans ta chambre" was forcefully accused. The lady also brought a man, for backup mayhap? In case of altercation? It was loud, and lasted for at least 10 mins. I was in my room wondering if now was a good time to get my cheese from the fridge. It sounded like they were just outside my door. I waited for the storm to pass.
- I almost got him evicted, but told property management to not do that to him because I was aware he had a pretty bad injury. I wish I made no attempts at kindness sometimes. Oh well it's over and given enough time this will become comedy.
Another roomate moved in the day the chain smoker left. She had a 70% proc rate per morning for vehement screaming. I don't know at who. Maybe family, because most people would just end a call from that much screaming.
- I love saving money…but sometimes that leads to situations with troubled youth who have no money. My incompetence in French prevents me from asking deep questions and being able to respond meaningfully.
- This is how I would awaken some days when she was there for 3 weeks. To screaming.
- Her voice would crack, and then she would try to hit higher decibels. She vaped and that affected my lungs and heart too, but was mostly out of the apartment until night time. She was otherwise nice to me. She told me she was on social assistance. I'm glad France takes care of its citizens.
- She told me about how she wanted to move to the city we were in and she was from Dardogne, where people were not nice. I sensed a suggestion of violence and harassment, but had no idea how to express that up to Unit 4 in Duolingo's French course.
Did two rounds of job applications to US companies between Aug 1 and Aug 14, while unable to breathe at my favored capacity, sleeping on the floor, and having loud interruptions.
Surprisingly, interviewed for a job between presenting lectures from my room in France, amidst the chaos woven with domestic peace.
- Met with hiring manager
- Met with someone who gave me more of a code design assessment
- Got to onsite round and learned they'd ask a system design round.
Scrambled to cram system design that I learn is happening the next day from the recruiter who sent out the email. I pushed aside all else and reviewed by
- rewatching some videos / rereading on designing a stock exchange
- Review what makes cassandra so great
- Review leaderless replication
Gave meaningful feedback to one of my student's for his presentation on Union Find.
Continued practicing leetcode.
- Though I had to look up the answer for one of the daily problems and straight up and almost copy it. I put in the work to understand why my recurrence relation was incorrect, found a minimal test case, and saw I didn't actually try all valid subproblems, and saw how the model solution handled it.
- My streak means consistent effort, and most days I get the problem without looking up a solution but may use hints. I try to limit myself to 2 hours of consistent effort before looking at the editorial. This month was harder than every month prior this year. I didn't give up on the hard that ended this month and it took me 4+ hours and plenty of wrong attempts to get a solve. I would have had no chance had I not reviewed Dijkstra earlier.
Lingering in France sounded good, in part because I don't have enough time to learn Spanish. I found a new place, cleaned my unit, picked up my bags.
- Hey, a pound of brie is < $4. I think there must be a government regulation + subsidy, because no way can it be this cheap. That's enough to change anyone's mind to stay here just a little while longer.
- There are of course, more expensive bries, and they are also great.
I'm really good at looking lost and asking for directions somehow.
- I think it's how I preface by asking if they speak English (they'll say no), then I proceed to try my best in French, which is usually enough.
- A cadre of guffawing hommes were standing outside the train station and helped me find the right bus. It turns out the guffawing hommes were all bus drivers, and one of them led me to the bus I was going to almost vomit in.
I somehow mentally jujutsu'd myself to not vomit in a bus; it would have been too embarassing.
- We rode through wind after wind of a tortuous mountain pass. I hadn't eaten enough in the morning because I was cleaning and clearing out.
- I was worried I wouldn't be able to explain myself to anyone if I vomited during the ride. I walked up to the bus driver when we stopped at a stop and let him know I had "mal à la voiture" (thanks google translate). And sat up front, which helped. I think the other people on the ride were confused.
- It was agony. Feeling on the edge of vomiting for over an hour, clutching a plastic bag and sweating all over it as the nerves in my fingers tingled. Maybe I was hyperventilating too. I was nauseous.
- That's how the Auvergne-Rhône-Alpes region do, an earthen furrowed brow.
Sadly forgot my adapter at the place before the agony of vomit–so it was time to problem solve. If my laptop runs out of battery I won't be able to continue my leetcode streak.
- I texted my current host where I could buy a new adapter. After visiting a store and not finding it, he unexpectedly offered to drive me to a store to find an adapter the same day.
- Problem solved at this place called Darty. "Solvé," I heard my host say. He then dropped me off at a bakery and I bought some good bread. Walked back up a mountain. Yay. A good kind of strenuous. Still need to see a doctor one day so I can't overdo it.
Surprisingly am closer than ever to a job I had a call with the hiring manager who said "You had a strong technical, and a strong behavioral," and that "he wish he could send me the offer letter now." So…it's probably happening but I haven't heard word since the 28th. Until I start, I'm assuming I still don't have anything. Companies may still rescind offers.
Allow the world to show me what's next. Being open to possibility, meaning, and connection.
For the first time in a long time, I'm allowing myself peace. It's been hard since last September. Very little did I look forward to.

Meaningful losses:

The loss of time due to my lack of focus. It is something that cannot be recovered. (I don't think this was a character error on my part; uncomfortable living situations are just hard to manage).
Didn't:
- Put ob-libre-translate on melpa
- write more blog articles
- go to that meetup where French people spoke English. Missed it 3 times because they were all on interview days, or days before interview days.
Like last month, I've lost lots of sleep. I think I can start recovering here though.

What I'm grateful for:

My students, I mentioned already he's presenting on Union Find! He's going to show off its classical use in Kruskal's algorithm as well as applications to leetcode problems.
The chance to interview at that one company.
The injection of insecurity from that "Design S3" system design interview I died on last year. It will only make me stronger.
This star log as I careen through abyssal space-time. The hiring manager looked at this journal. Apparently this gave me some cred. Huh. It's almost like my plan has at least marginal success. (yay)
France. No, the people are not friendly a lot of times, and a lot of them are not helpful, but there are always those that are. I am thankful for this country for not putting up barriers to my entry, and allowing me to repose.
The cost of living in France inspires the will to live. A man can be free here, unburdened by the orphan crushing machine of Les États-Units.
The kindness of others suggesting I look abroad to live for a bit.
My friend who kept my things safe for me in the Bay Area
My Airbnb host in France.

Looking ahead:

Still on the docket: Attempt to seriously write articles more.
May need to find a place to stay in the US now.
For the contacts I've collected, I know I need to help some people.
Think about setting up some resources or time aside to help the emacs community.
Continue Duolingo to improve French. Meetup is not popular in France so I will
Oddly enough, I didn't do any AI or Godot learning this month, like I thought I might this month. And the strategy turned out to be finding work, and
Finding my way back to the US.
I may know an old lady in France who might let me stay at her place if I work on her garden. Not really sure how that will play out, or if it will at all.

4024-07 In Review

4024-07-30T00:00:00Z

My meaningful achievements this month:

Couch surfed at a stranger's house! and it went well. And it was comfortable.

Organized and led 3 algorithms lectures, one of which was in-person.
- It would be 4, but my flight overlapped on that day, and I needed time to settle in.
- Went over Dijkstra's algorithm, and it's common implementation versus how it's traditionally taught with the decrease_key operation in a priority queue. I still need to edit that video.
Went to an Emacs SF meetup, and suprisingly, met someone and got their number without realizing I had been admiring their work in the Emacs community for many years!
- I wish I had more in common to start a fire or was staying longer to match on more interests.
Submitted a packet for the Onion fellowship and I know I won't get selected. I got good feedback from a friend who said to keep things shorter who probably will get selected, and I'll keep that in mind. I'm glad I tried this writing exercise.
Came (a bit begrudgingly) to France, purely for cost of living savings. Not to have fun. Not to see the Olympics.
- It turns out my worries about needing proof of subsistence weren't necessary. A US Passport gives you the benefit of the doubt in so many ways. I didn't need to produce any documentation about my stay. I don't even have a return flight.
- Schwab was down and I couldn't move any money from the bungled Crowdstrike deploy (who shipped to 100% of users without doing canary deploys or testing? What a company.). I was worried about being denied entry due to lack of funds in my bank account.
Second Place at an AI and Security themed hackathon
- And I left before any winning was announced because I had to meet with a person I was couch surfing with.
Networked with people at the hackathon
- But I need to go deeper! I need to help them outside of the hackathon.
- A couple people told me I should write on medium.
- I also didn't capitalize and really meet with as many people as I'd like, but also my team needed me. The hackathon was chaos.
Helped someone, ever-so-slightly, understand just a bit more about lisp macros. When they asked, I had pretty bad diarrhea at a public library when the only stall was occupied.
Took the TGV down to southern France. It's still really busy here though, lots of people here for the Olympic games.
Survived some kind of diarrhea that plagued me for days.
Not dying; not being sicker; not getting anyone else sick.
<4024-08-10 Sat> Updates to this month:
- Wrote ob-libre-translate, my first emacs package that I wrote on the 31st, after this article. It's a small one, and I had a template from ob-translate, but still; yay. I tried writing an article on it, but didn't finish in this month. I still need a piece of creative writing for it, and an image.
- Applied to some jobs this month. It's a slog to do that. And I secured one interview! Wow the chances are usually < 1% for each job application.

Meaningful losses:

Didn't hit month goal of 3 articles this month. This one counts. I'm missing 2 others.
Came to France, leaving my life in America at a pause. I enjoyed the Bay Area and part of me didn't want to leave. I had roots here, until I uprooted myself more than a decade ago. Am I really able to network in France?
Someone I know didn't have time to meet during the in-person seminar. I likely won't see her for a while. This didn't hurt emotionally, but it was a meaningful loss because I could have helped a student network for a job. (And maybe network for a job myself).
I've lost lots of sleep. In addition I was hauling ~50lbs in hot weather and not eating much. The body has tiredness.
The place where I'm staying unsettles me because of
- noise; there's 4 other people here. And lots of banging up stairs.
- people smoking indoors
- the place is dirty; I found a cigarette bud on the kitchen floor. It's a total guy-sty. Things feel greasy when I touch them.
- and I paid for a month. So I'm locked in…
My friend totally botched scheduling after I told him for months I needed a ride from Bay Area TO Seattle. He kept thinking I was going the other way, and our roadtrip is effectively canceled, and I had to scramble for housing, kind of why I unexpectedly needed to travel ASAP.
Logistics eat so much time it's effectively an entire project to just get from place to place. I'm grateful I can travel, but still.
Got sick in the Bay Area. Not sure why. I got better just hours before getting on that flight to Paris, which was too close.

What I'm grateful for:

My students. They've connected me with the hackathon, which led to some connections in the Bay Area. I also got a free meal from one of my students and an apple.
My friend who teaches math took me out for meals and I got to hear about the girl he's dating.
- Put away a suitcase in his storage. That suitcase is half my life right now. Glad it's safe and I'm not lugging it around.
The host on couch surf. He let me do whatever I needed to do and didn't need me to hang out with him. I used his cooking utensils and his bed was cool during these summer nights.
Oakland in general. I didn't realize there were such cool spots to hang out. I was too poor back then and trapped in violence to really see anything else than the people I wanted to hurt.
AC transit. Unlike the olden days, I think AC transit has AC now. 51B was cool during a hot day.
The shopkeeper in Paris from whom I bought 30 eggs from. I then proceeded to hardboil 30 eggs and packed them in a ziplock bag. They were good snacks since I hadn't been eating much on the international flight; I also ate some on the TGV the next day. I'm going to eat my 30th egg from that carton now.
The French cost of living! Buying weird cheeses that have some mold on them for a discount from the "anti-gaspi" section of the "hypermarket."
Peaches/nectarines are in season (and they're cheap!).
The shopkeeper at Spar who helped me find laundry detergent and was amused by my kind of broken French.
France doesn't really have air conditioning, and despite my grievances about my living space, it's cool enough to sleep. I'm sleeping on the ground though instead of on the bed (it's complicated).
A friend from high school in the Bay Area let me stay with her and her husband. They had a really friendly cat that reminded me of my college days. Her husband also makes really good eggs and is a good chef in general.
Another friend from high school told me about a job opening at his friend's place. We went out to lunch in Oakland at a nice Thai restaurant. I enjoyed hearing about his life now. He's going to be a dad!
Another friend from college talked to her parents and I got to stay at her parents' place. They cooked good food and I told them also that they needn't cook for me. They'll welcome me back in the future too!
Some guy stole a dozen eggs from me at Berkeley Public Library as I was using the restroom on the 2nd floor. I found him. Chased him. And I realized after seeing him, he's in a much worse spot than me. I'm glad I let that go. He better enjoy those eggs!
Okay, I think I kind of carried my team in terms of direction and helping everyone, but my hackathon teammates were cool people too.
Airplane bartering episode: I don't eat meat so I gave my in-flight meal to a guy in another aisle who then gave me double-stuffed oreos which I then gave to the French guy sitting next to me who gave me his thing of chickpeas earlier. And the other guy sitting next to me who tapped the other-aisle-guy on the shoulder so I could initiate. If only I could network like that for jobs.

Looking ahead:

Wrap up the algorithms seminar, covering up to either a first or second lecture on Dynamic Programming.
Attempt to seriously write articles more.
May try to couch surf more in this town, or nearby towns. I'm going to sleep at weird hours because it's so noisy. I think children run up and down upstairs. Lots of banging and sometimes wall shaking.
I've collected contacts…but I need to somehow create value for people here. Connect others.
Where do I go from here? Berlin, Germany? Try to network in Paris? Lisbon, Portugal? Malaga, Spain?
Je veux améliorer mon français.
Either I study AI, or dedicate time to learning 3D Godot in August. I need to somehow find an opening in the job market, or create something to attract income.
If I don't do either of the above, it'll most likely because of a software project I'll undertake. I don't know what that would be yet.
Apply for jobs in the US. I want to return… but I might be able to chill in SE Asia for cheap after my stay in Europe. I don't like flying because of carbon emissions. If I had not failed so hard in acquiring income I would be sitting in Seattle.
I may know an old lady in France who might let me stay at her place if I work on her garden. Not really sure how that will play out, or if it will at all.

4024-06 In Review

4024-06-30T00:00:00Z

My meaningful achievements this month:

Organized and led 5 algorithms lectures, once each Saturday. I tallied my time. One lecture takes 10-16 hours of my time per week. I make slides in emacs, curate problems for the lecture, find real life applications, (re)solve problems so I can decide whether they're suitable, and dry-run my lectures.
- I have recorded lectures, but I only uploaded the most recent one to youtube so far. Yay binary search. I definitely had a choke moment. Wish it was shorter…it's actually in two parts.
- Getting started took a lot of time too: deciding on the technology to make slides, rendering to PDF, fixing bugs/tweaking things in the export.
- I also have been doing individual reach outs to students but I haven't gotten much of a response.
Started a book club and finished a book. And genuinely, I think the people that participated got something out of it.
- It was Supercommunicators by Charles Duhigg
Went to an open mic, bombed, and met people that I later talked with outside of the event. Got 5 people's numbers, and I irregularly talk to 2 of those people now! Connected with one person in tech on linkedin but he isn't responsive.
- A complete stranger in the crowd went out of her way to tell me they enjoyed me going up there (they were being nice, I had no material).
- Got invited to another event by someone I met there, to attend a lesson on standup in LA. I ended up going and was able to listen and give my time and attention meaningfully to others.
Wrote a prototype for a game based off of someone's idea from the open mic. Met up with them to show them. Made short video to show them what I built. I'm glad I learned enough Godot last year to just execute. This could be a deadend, but building it was cool anyway.
Published the first npm package of a sequence heap, added CI, linting, coverage, tests, creative writing, and images. I had working code already 5-6 months ago, and all the things outside of pure execution took more than 2x the time. Learned enough Krita to do some image editing using masks and different brushes. Storyboarding the gif and the subsequent video took a nonnegligible amount of time as well.
Learned enough Krita to make a simple gif for the sequence-heap project. Embarassingly, it took me a long time to make the proofs of time complexity.
Learned enough KDEnlive to do the video editing to explain the sequence heap. Video editing took an immense amount of time. Re-recording is painful. I also don't have a GPU to help me out when rendering.
- Had a huge headache because the way it was built for my OS made rendering not work. So I had to find a specific older version to use.
Uploaded my first youtube video on a sequence heap! I sped it up so the speed is 1.2x because I felt I talked too slowly. It might say it was published today, but I had it unlisted for a while.
Published 2 articles on this blog this month (this is the second one!).

Meaningful losses:

A lot of hair ☹️.
A relationship I don't want to repair; a living situation where I am increasingly uncomfortable.
Lost budding relationships with people because I cannot reach them. A car-centric way of life is Southern California. I cannot build familiarity with other people with my lack of mobility. I am absolutely sure I can benefit others, but that means little if I do not show up.

Looking ahead:

Open myself to couch surfing around other people's places. A friend thinks I can use this situation to network. I am not so optimistic.
Or leave America. And live somewhere cheaper for a while. Still unemployed. There is no one who will help me until I can bring enough value to them.
Do an in-person seminar at a community college in July. Two people are interested. I do want to connect better with students, and help them where they are at.
Continue lectures every Saturday in July even if no one shows up. I can at least practice my communication skills, video/audio editing skills, speaking, and problem solving skills. When I will one day need these skills for something else, I can show up for those situations.

Supercommunicators, Book Club

4024-06-05T00:00:00Z

I've found others interested in the art of human connection and pitched the idea of doing a book club on Supercommunicators with me. Success!

I've been a fan of Charles Duhigg's work since I read The Power of Habit over ten years ago, back when I first started reading about how to get more done.

I already read the book, and now I can learn from others and provide value at the same time.

Here's a 4 week agenda I made. Feel free to use it, share it, or suggest improvements!

Week 1:

83 pages

Prologue (pp i-xviii)
The Three Kinds of Conversation (pp. 1-33)
The What's This Really About? Conversation (pp. 35-75)

Questions for Reading Comprehension:

What do you hope to gain from this section?
What are you hoping to get out of the book and this book club?
What is the goal of a conversation?
Finish this sentence: C_ _ _ _ _ _ _ _ _ _ _ _ is connection.
Explain in your own words the Matching Principle and how it's different than mimicking.
What is neural entrainment? Describe a moment in your life where you felt neurally entrained.
Most recently, when did you not follow Rule One and not pay attention to the conversation? What happened?
What did you learn from this week's reading?

Discussion topics:

What kind of conversation are we having now?
What did you learn from this week's reading?
I'm here because I want to be more consistent as a communicator. What about you?
Has anyone prepared for conversations? How did it work out?
Has anyone tried meetings where the cultural norm, or rule was to declared what their goal for the conversation was? How did it go?
Boly's creative perspective shifting impressed me. His ability to see Karl's values clearly allowed him to create a completely different argument to shift Karl's own perspective. When have you done something like this? Or do you have examples of someone else doing this?
Is there an upcoming conversation you want to prepare for? How will you prepare for it?

Week 2:

84 pages

The How Do We Feel? Conversation (pp. 77-161)

Questions for Reading Comprehension:

What do you hope to gain from this section?
Why did Epley stop drinking?
What 3 questions Epley ask the hedge fund managers to share?
How would you apply learnings from the Fast Friends Procedure to a non-laboratory social setting?
What role does vulnerability play in connecting with others?
How quickly can strangers connect?
Convert the following questions into deep questions, or describe how you would lead into a deep question:
- Do you like movies?
- What did you do this weekend?
- Where do you work?
How did McGuire select astronauts? What Supercommunicator principles did he follow?
How did writers of The Big Bang Theory end up having actors communicate emotions to an audience?
Name a conflict in your life. Why were you fighting? State its surface issue as well as the emotional conflict.
Reflect upon a time someone really connected with you. Did you loop for understanding?
What does controlling a conflict's boundaries mean to you?
What did you learn?

Discussion topics:

What did you learn?
What (new) ways do you want to bring the power of questions into your life?
I don't feel like Epley's experiment with hedge fund managers would make me feel more connected to other hedge fund managers. What about other people?
Tell me about how you handled a conflict recently. What did you do?
Does anyone have a conflict with people they care about and disagree with?
Has anyone calmed a storm by looping for understanding in an online forum? Or something close? How did you do it?

Week 3:

65 pages

The Who Are We? Conversation (pp. 169-234)

Questions for Reading Comprehension:

What do you hope to gain from this section?
In your own words, what is stereotype threat? Do you face this? What's a strategy you might use to mitigate stereotype threat?
What is the contact hypothesis?
How can you draw out multiple identities?
How does this chapter tie into what you know already about communicating during amidst conflict?
What does a Who Are We? Conversation look like? How do you know it succeeded?
Define motivational interviewing. Where else did the book mention this concept?
What is identity threat? How did this impact the culture at Netflix?
Think about a hard conversation you've had. What obstacles did you face? What can you do to overcome those obstacles in a similar conversation in the future?
How can you use the guidelines to help people you care about have an important conversations?
What did you learn?

Discussion topics:

What did you learn?
I'm really impressed with Chamie's ability to sway one of her patients. I'm also very surprised at Salma Mousa's experiment. What did you feel when reading those passages?
Are you trying to persuade someone currently? How can you use a Who Are We? conversation to help?

Week 4:

Free discussion; No pages, meet to talk about implementation and results or seek advice.

Solidifying understanding with review:

https://www.npr.org/2024/02/20/1196978652/life-kit-supercommunicators-charles-dughigg
Who do you consider a supercommunicator? What traits do they have?
How can someone else be sure you're not only waiting your turn to speak?
How is matching not the same as mimicking? Note a time in your life where you matched but did not mimic.

Discussion topics:

How is implementation going?
What other tactics do you use to connect with others that weren't mentioned?
What is your favorite way to connect with someone?
What did you disagree with in the book? Why?

Programming Helm: Quick File Access

4024-03-26T00:00:00Z

Code

(setq jwow/org-dir (expand-file-name "~/org/"))
;; Named these functions for debugging.
;; Wanted to modify the lists in place by adding the display name as first element
(defun jwow/helm-action  (candidate) (find-file (cl-first candidate)))
(defun jwow/helm-transformer (candidates)
  (mapcar
   (lambda (file-with-attributes)
     (cons (file-name-nondirectory (cl-first file-with-attributes))
           file-with-attributes))
   (sort candidates (lambda (c1 c2) (time-less-p (nth 7 c2) (nth 7 c1))))))

(setq jwow/org-dir-helm-source
      (helm-build-sync-source
          "Quick Open Org File"
        :action #'jwow/helm-action
        :candidates (directory-files-and-attributes jwow/org-dir
                                                    t
                                                    ".org$\\|.org.gpg$"
                                                    t)
        ;; sorted by last modified time, descending
        :candidate-transformer #'jwow/helm-transformer
        :diacritics t))

(defun jwow/org-dir-quick-jump ()
  (interactive)
  (helm :sources '(jwow/org-dir-helm-source)))

(define-key xah-fly-command-map (kbd "C-SPC o") #'jwow/org-dir-quick-jump)

Interprocess Communication (IPC) In Emacs for Typing Sounds

4024-01-30T00:00:00Z

CLACK CLACK CLACK. Do you hear that? It's the sound of an IBM Selectric typewriter. PING.

Making typewriter from emacs won't boost your productivity, but why the hell should it. Part of using emacs is making it your own.

But I ran into lag–my keystrokes struggled to get on the page–arresting my evil schemes to a full stop. Every future propaganda leader needs the ability to propagate their ideas. It was unacceptable!

In this article, I investigate quick ways to make the selectric-mode package faster, from trying emacs threads to an mpv daemon.

If you just want to look at results, skip to the Full Code or Performance comparison.

Since I originally wrote this article (unpublished) on January of the same solar year, I've come up with yet another solution.

I'll also try to upload a video of this at some point.

This blog post also has a sequel that I'll upload soon! – 2024-12-24

Here it is – 2024-01-09

Prerequisites

mpv installed
- mpv will essentially act as a daemon that we can write text to and return control quickly to emacs
*nix operating system
- (this might work on windows, untested; windows works a bit differently with pipes)
An audio file
(optional) emacs selectric-mode

Why use IPC for this?

Emacs is mainly single threaded¹. So even though emacs has (play-sound), this will block, locking up emacs for the duration of the sound.

Much of parallelism in emacs boils down to forking a process and sending commands to it.

That's what selectric-mode does. It calls

(start-process "*Messages*" nil "aplay" sound-file-name)

And hey, it works. Play a sound a file from a different process and the user is back in control in emacs.

But forking processes is expensive. It costs both time, and memory. What's more–we're just playing the same sound. Do we really need to start a process again and again?

My computer couldn't handle selectric very well when I first used it years ago. I felt this when I was using a selectric-mode on a pentium core processor….It's runs okay now, but my computer still gets very hot. No one has worked on the performance of it and the author has not responded to issues.

Looking up playing audio online with emacs shows people mostly using aplay or play however.

Couldn't we just decouple the caller (emacs) from the receiver (thing that plays audio), and keep the receiver alive?

Thread Detour

Before diving into the daemon approach, I wanted to try threads.

So instead of (start-process ...) on the main thread, we can instead move that off to another thread. This frees up the command loop² from starting the process.

(use-package selectric-mode
  :config
  (defvar jwow/typewriter-enabled nil)
  (defun jwow/threaded-sound () (make-thread #'selectric-move))
  (defun jwow/toggle-typewriter ()
    (interactive)
    (if (setq jwow/typewriter-enabled (not jwow/typewriter-enabled))
        (add-hook 'post-command-hook #'jwow/threaded-sound)
      (remove-hook 'post-command-hook #'jwow/threaded-sound)))

  ;; Now bind the key to a keymap, since I use xfk I bound it like so
  ;; (define-key xah-fly-leader-key-map  (kbd "; b") #'jwow/toggle-typewriter)
)

M-x jwow/toggle-typewriter to toggle.

This solves the blocking issue. I was able to type continuously, holding down a key for 30 seconds without lag.

But the CPU was still high. See Performance comparison.

That's because starting a process each time was expensive. What would help was a long running process with the audio already loaded in memory. Something we could control, like mpv.

MPV Approach

The idea is taken from wasamasa's chip8.el: use mpv as a daemon and feed it messages from emacs. Emacs regains control and does other logic while the audio plays.

I first heard about this approach from his emacsconf talk. I tried the code, but it doesn't work anymore because mpv dropped the --input-file command option.

Eventually I stumbled onto other flags, like --input-ipc-server. And the mpv package, empv.el, takes a similar approach in creating the socket now. In my case it's a unix socket file.

All that's left is for emacs to send messages through the socket. Of course, emacs has a native API for that: Network Processes.

Once we have this process that acts as a wrapper for the socket, we simply talk to the process with (process-send-string process-reference string). Would it work?

Code, step by step

First, get selectric-mode

Most readers by now will be familiar with use-package. If you have melpa as a repository you can simply run:

(use-package selectric-mode)
(selectric-mode 1)

Run mpv with a socket

Now let's get a socket so we can remote control mpv

(setq jwow/mpv-audio-file (let ((selectric-audio-file
                                 (expand-file-name
                                  (format "%s/../selectric-move.wav"
                                          (find-library-name "selectric-mode")))))
                            (message selectric-audio-file)))
(setq jwow/unix-socket-file (expand-file-name "~/ipc-socket"))

;; This is the process
(defvar mpv-process)
(defun jwow/start-mpv-with-socket (socket-file audio-file)
  (setq mpv-process
        ;; Note that if you have a config, that might have different settings than how the process is started The
        ;; options are more verbose here because I have defaults that I'm overriding
        (start-process "mpv" "*mpv*" "mpv"
                       "--no-video" "--loop-playlist=no" "--pause"
                       ;; Otherwise the *mpv* buffer gets really full 
                       "--no-terminal"
                       "--keep-open" (format "--input-ipc-server=%s" socket-file)
                       audio-file)))

(jwow/start-mpv-with-socket jwow/unix-socket-file jwow/mpv-audio-file)

Finally, connect to the socket

Note the use of make-network-process. This is how we wrap the socket. By the time I was done with this code and looked at empv.el

;; This must be run after the mpv process has started
;; This is because we connect to that process
(setq jwow/socket-process 
      (make-network-process
       :name "socket-client"
       :buffer "*socket-output*"
       ;; Ensure that we are talking through a unix socket using 'local
       :family 'local
       ;; Usually this is a protocol name but 
       :service jwow/unix-socket-file))

(defun selectric-type-sound ()
  ;; The \n is important, otherwise mpv will not recognize the command. Commands apparently must be terminated with a new line.
  (process-send-string jwow/socket-process "seek -10; cycle pause\n"))

Performance comparison

For this section, I just looked at the graph from gnome system monitor. I also didn't type for the same duration in each of the three approaches as there was enough data.

Without any sound

Figure 1: 15% CPU normally

Using plain selectric mode:

Figure 2: 55-60% CPU without changes

Using threads:

Figure 3: 40% CPU using threads

Using mpv:

Figure 4: 20% CPU using mpv

Although the screenshots are from different days, and therefore slightly different state on my computer, these results tend to hold regardless. I was happy enough with my approach, but I won't pretend it's perfect.

Issues in the mpv approach

What if you wanted to play multiple files at once?

An mpv instance can only play one audio file at a time (as far as I know). This either means you spin up multiple processes of mpv, or you find some kind of process that can do audio mixing. Like a synth, or a DAW which uses synths. (Sorry if I mess up the technical terms, audio experts)

I don't know too much about audio so maybe there's a lightweight way to do this. For what it's worth, something mixes down everything in realtime into ALSA, so there's probably a way.

Audio starts and stops

The two commands sent to selectric are seek -10; cycle pause\n. In the middle of playing, mpv will seek back 10 seconds, meaning it stops playing the current file.

This stems from the lack of mixing functionality.

Type sound is too loud

Sounds like a gun. The sound files themselves may need to be modified.

Can we get the CPU lower?

Making a sound doesn't seem like it should cost so much.

Exploring more

In 2024-12-24, I got excited by another general solution. Join me there.
mpv has many options and you can drive it completely by ipc (I think!). Try grabbing data from the mpv process too!
empv.el drives mpv using ipc as well
If the process you want to communicate with registers itself with D-Bus, you can try emacs's D-Bus API. To debug and find message signatures, I've had good luck with the D-Spy tool. Things are a bit cryptic since many programs have lacking documentation on how they are controlled via dbus.
In the example here, I only changed the execution flow when when a user was typing. I didn't change how the move sound worked in the typewriter. You can actually try other commands to drive mpv remotely here: https://github.com/mpv-player/mpv/blob/master/DOCS/man/input.rst
- It'd be interesting to try loadfile <url> [<flags> [<index> [<options>]]] to load other files into mpv and cycle through them. Would it be performant?
I tried out midio which may do what you're looking for. From my understanding, you send midi commands to fluidsynth, which is kind of similar to what we're doing here.

Full code

Copy and paste into a scratch buffer and M-x eval-buffer!

;; I'm using selectric mode, but you can try out the code without it. 
;; Just make sure you have an audio file you can use
;; And call (process-send-string jwow/socket-process "seek -10; cycle pause\n")
(use-package selectric-mode)
(selectric-mode 1)
(setq jwow/mpv-audio-file (let ((selectric-audio-file
                                 (expand-file-name
                                  (format "%s/../selectric-move.wav"
                                          (find-library-name "selectric-mode")))))
                            (message selectric-audio-file)))
(setq jwow/unix-socket-file (expand-file-name "~/ipc-socket"))

;; This is the process
(defvar mpv-process)
(defun jwow/start-mpv-with-socket (socket-file audio-file)
  (setq mpv-process
        ;; Note that if you have a config, that might have different settings than how the process is started The
        ;; options are more verbose here because I have defaults that I'm overriding
        (start-process "mpv" "*mpv*" "mpv"
                       "--no-video" "--loop-playlist=no" "--pause"
                       ;; Otherwise the *mpv* buffer gets really full 
                       "--no-terminal"
                       "--keep-open" (format "--input-ipc-server=%s" socket-file)
                       audio-file)))

(jwow/start-mpv-with-socket jwow/unix-socket-file jwow/mpv-audio-file)
;; This must be run after the mpv process has started
;; This is because we connect to that process
(setq jwow/socket-process 
      (make-network-process
       :name "socket-client"
       :buffer "*socket-output*"
       ;; Ensure that we are talking through a unix socket using 'local
       :family 'local
       ;; Usually this is a protocol name but 
       :service jwow/unix-socket-file))

(defun selectric-type-sound ()
  ;; The \n is important, otherwise mpv will not recognize the command. Commands apparently must be terminated with a new line.
  (process-send-string jwow/socket-process "seek -10; cycle pause\n"))

Footnotes:

Threading capabilities have been added actually elisp#Threads.

The command loop is what gives you the feeling of control–it listens to your input. If this main thread was busy doing X for a while, you wouldn't be able to get your next keystroke on the buffer until emacs finished doing X.