I have a folder marked executable with hundreds of small shell scripts, appended to $PATH. I never write long scripts. I sometimes write scripts that run other scripts but I try to avoid such dependencies. I do not use bash, I use NetBSD Almquist or modified Debian Almquist shell so all the scripts are highly portable. I can move a tarball of this folder from computer to computer, whether the OS is Linux or BSD. These scripts generally run faster than bash scripts.

It would be untenable to save all command line history since I spend every day in the shell in textmode (not x11) and only occasionally switch to a graphical environment. I sometimes use Almquist-based shells that have command line history removed. This forces me to write useful scripts.

A lot of cumulative shell knowledge is contained in that folder of scripts. These scripts have worked for decades unchanged. I think the Almquist shell may be my favourite interpreter. No matter what else I try I always come back to the shell language. There is something appealing about scripts that seem to work forever.

  ⟩ history | wc -l
  170919 

Your hardware might be fast enough that it's negligible, but several years ago (maybe 2018-19ish?) I noticed my shell startup was getting sluggish and traced a large fraction of it back to the time bash took to load many lines into history.

I still like keeping it all, so I wrote something to dump it into a database as I go but then truncate working histories to something shorter (500, I think).

  $ time zsh -i -c 'exit'
  zsh -i -c 'exit'  0.03s user 0.02s system 100% cpu 0.055 total

    time zsh -i -c 'exit'
    zsh -i -c 'exit'  0.33s user 0.31s system 97% cpu 0.658 total

* Load asynchronously

* Only have a barrier if I reverse-search.

Might be over-engineered, though.

  $ time zsh
  $ ^D
  zsh  0.14s user 0.04s system 80% cpu 0.219 total

  time zsh --login -c 'logout'

See zshbuiltins(1).

  $ time zsh --login -c 'logout'
  zsh --login -c 'logout'  0.03s user 0.01s system 99% cpu 0.037 total

This was my experience as well. The --login flag was recommended in order to address concerns raised by @abathur.

> Whenever I've had noticeably slow zsh startup times in the past, it was almost always some plugin/extension doing something very dumb (e.g. stuff like full 'git status' in a large repo -- just takes time); not the history management.

Great point.

Unfortunately, my limited research suggests tracking down which plugin or extension is the root cause is a manual effort starting with the contents of the canonical zsh initialization files (often named .zlogin, .zprofile, .zshenv, and .zshrc).

i think you can debug-trace loading time putting some commands on your zshconfig

I mean, this might seem really bizarre to worry about, but some folks use tiling window managers and just expect to immediately start typing once the “new terminal” key is hit…

Actually this has been slightly annoying me lately after making my system pretty. I added some fancy compositor stuff and now if I do “open terminal,” “exit” the computer complains that it doesn’t know what xit means. Not a big enough problem to fix though.

Edit: huh, actually playing with it this seems to be 99% the fault of the terminal emulator anyway, not the shell or my silly special effects.

Unless the system is under memory pressure, most shell initialization will read from in-memory OS file caches and not be noticeable as you note.

Where significant delays are often seen is when a seemingly innocuous extension uses network-based or some other heavy file system I/O commands (such as a "find $HOME -type f" type of thing).

Unfortunately I've hit some obscure bug with exactly when `HISTFILESIZE` is applied, and had some shells truncate all my history undesirably. There are also problems with reading history if the timestamp format changes.

I did anything at all because the default shell profiles in macOS can cause history loss, and I'd found my last straw. I put them in sqlite because--if I was bothering to build something bespoke--I wanted to track more command-time context to build tooling around later. (Including to play with some ~curation ideas.)

I might have decided to just use Atuin if it existed at the time, but I did it three years and change before its first release. (There were maybe 5 or 6 barely-used public examples of this idea on GH at the time, but none tracked everything I wanted among other issues.)

  wc -l .zhistory
  121104 .zhistory

I use bash within tmux heavily, and got irked that a command I run in one shell session is not immediately available as a history item in other concurrent shell sessions. So I wrote a history plugin based on bash-preexec to track everything to two files: a per-directory history file, and a global history file.

I have a bash function which does history selection for me, by popping an fzf selector to look at the directory-specific history file. A keybinding within fzf allows me to switch to looking at the global history file instead.

Boring commands such as “cd”, “pushd”, and a few others don’t get logged. The log entries are in json format and include basic metadata; directory, timestamp, and pid.

Within the fzf history picker, another keybinding allows me to edit whichever file I’m actively using. So if I fumble a few times to construct a command, then when I get it right I just pop into the selector, edit the relevant file, and remove the lines I don’t want to misfire on again.

I’m sure this is basically what atuin does; now that I’m at the spot where directories are the unit of history relevant history, maybe I should give that tool another look.

One really interesting upside of all of this is that I now tend to make “activity-specific” directories in my repos. For example, I have a “.deploy” directory at the git root of most of my projects. There are no files within that directory; but my tool creates ~/.bash.d/history/home/belden/github/company/project/.deploy.json which contains the history of ~/github/company/project/.deploy/

Empty directories are invisible to git, but for me the directory “has” content: the log of how I need to deploy this service or that service.

It’s a weird way to use my shell, and just sprung out of the initial grief: I shouldn’t have to exit a shell session to have its history become available.

this can be solved in vanilla bash by adding these lines to .bashrc (be sure to remove other HIST* configs options from it)

    shopt -s histappend
    export HISTSIZE=100000
    export HISTFILESIZE=100000
    export HISTTIMEFORMAT="%F %T "
    export HISTCONTROL=ignoredups:erasedups
    export PROMPT_COMMAND="history -a;history -c;history -r;$PROMPT_COMMAND"

It's hard to understand how any other way is usable. If you have a dozen concurrent shell sessions, some running a long running process for example, and you had to restart one of them, you wouldn't be able to just Ctrl-C and go to the previous command. For me it's way better shell sessions live their own life but the history is accumulated to a single pool.

Lately I have been using atuin for this. I have some gripes about interface defaults, and how the interactive search for history wipes out most of what was on my terminal view and causes a big "wtf was I doing again?" moment when it changes the terminal so much, but the tradeoff of shared history is very very much worth it for me.

AI Agents using tools also benefit from seeing past usage of the tools (what worked, what didnt) to help them inform future usage

I was thinking about this the other day and I was specifically trying to think of a way to avoid the command typos ending up in the history file. I don’t think it’s useful to keep those around.

But yea maybe removing them from history so dont they auto-comlete is a good call

People might find that weird, and it sure is a tad inconvenient when random, low-priority websites cosplay Internet Fort Knox with very short and annoying MFA login timeouts and methods, but for me, the benefits (I'm less susceptible to tracking and can click on stuff with less angst that I might somehow leak data from my nigh-eternal browser session to some site or another, and I get to start relatively fresh in each session) outweigh the costs.

I don't do this, but I do disable my browser history on my work laptop so that I'm forced to author things I, and presumably other people, can actually stumble their way to later on.

https://hiandrewquinn.github.io/til-site/posts/disable-your-...

I also spend most of my online life in incognito windows, at least for sites that don't absolutely require a login. This keeps my browser history clean from all the disposable pages that I only visit once, and I take care to do only the more meaningful stuff in a regular browser window.

  HISTCONTROL=ignoreboth
  HISTIGNORE='rm *:ls *:cd *:cp *:builtin *'

    (cd ~ && [ -d ~/backup_mnt/ ] && rsync    -xav --delete --exclude='backup_mnt' --exclude='.debug' --exclude='.cache' "$HOME" ~/backup_mnt/backup )   # rsync DELETE
    (cd ~ && [ -d ~/backup_mnt/ ] && rsync    -xav          --exclude='backup_mnt' --exclude='.debug' --exclude='.cache' "$HOME" ~/backup_mnt/backup )   # rsync NO DELETE
    (cd ~ && [ -d ~/backup_mnt/ ] && rsync -n -xav          --exclude='backup_mnt' --exclude='.debug' --exclude='.cache' "$HOME" ~/backup_mnt/backup )   # rsync LIST

When trying to modify a command, I remove the space, and when done I search for saving history and editing with '/_hi':

    history -a ; vi ~/.bash_history && history -r

I also keep the similar lines aligned with spacing to note the differences quickly. (e.g. the -n above)

When exiting a shell and there's nothing interesting, history -r; exit

Also always I go to the end of the file before quitting vi so next time I'm where it last ended and before the appended new commands.

It takes work upfront and discipline but the everyday use of shell becomes very fast and clean.

HISTIGNORE='m:??:info STARHERE:info:[bf]g:exit:[bf]g %[0-9]:help STARHERE:date:cal:cal ????:exec env ENV\STARHERE' (replace STARHERE with an asterisk)

generally works for me in bash(1).

The line above set up automatic history ignoring for the colon-delimited shell globs; eg

fg %3

won’t be recorded to shell history, since it’s matched by one of the globs.

  HISTIGNORE='m:??:info *:info:[bf]g:exit:[bf]g %[0-9]:help *:date:cal:cal ????:exec env ENV\*'

You can optionally sync your history to a server in encrypted form to keep a shared history across hosts. The server is extremely easy to self host.

As discussed in TFA, it’s easy to filter or scrub junk like `cd ~/Desktop` so it doesn’t pollute your history. You can also fuzzy search and toggle between commands run in your current session, commands run on your current host, commands run in your CWD, and commands run on all hosts.

It’s my single favorite piece of dev tooling, and has made my job + life far smoother and easier. Highly recommend.

https://atuin.sh/

considering to ask where my json bash history is located and how that works - I noticed Ctrl-r looked different a while ago.

There's no way I'm going to manually curate every command I type, I've got actual work to do.

Anything important and convoluted, I'll shove it in the fish config for quick access.

I don't do this because I didn't know "unset HISTFILE" was an option. Instead I symlink .bash_history (and other history things like .python_history, lesshst, etc.) to /dev/null

* Looking back at my old process for onboarding into some new technology. I can easily see how my technique evolved as I optimized for more correctness (e.g. cp -r vs cp -rp vs rsync with whatever flags does everything I want, I know I want the most recent incantation)

* Figure out the exact process I did for a backup or other movement of data when I want to ensure I have copied X, Y, and Z. Sometimes my shell history has saved me where my notes have failed.

99% of the time, I never look at it, but when I do need to look at it, it has been great. My boss once asked me: "What args and screening file did we use when we made that one-off DB 4 months ago?" Was able to check and confirm it was correct. Or for personal use: "Where did I move that folder of pictures?"

For the curious, the salient .bashrc bits are:

  function _setAndReloadHistory {
      builtin history -a
      builtin history -c
      builtin history -r
  }
  
  # preserve shell history
  set -o history
  # preserve multiline commands...
  shopt -s cmdhist
  # preserve multiline command as literally as possible
  shopt -s lithist
  # reedit failed history substitutions
  shopt -s histreedit
  # enforce careful mode... we'll see how this goes
  shopt -s histverify
  
  # timestamp all history entries
  HISTTIMEFORMAT='%Y-%m-%d:%H:%M '
  
  # not the default, we like to be explicit that we are not using defaults
  HISTFILE=~/.bash_eternal_history
  # preserve all history, forever
  HISTSIZE=-1
  # preserve all history, forever, on disk
  HISTFILESIZE=-1
  # record only one instance of a command repeated after itself
  HISTCONTROL=ignoredups
  
  # preserve history across/between shell sessions...
  # ...when the shell exits...
  shopt -s histappend
  # ...and after every command...
  PROMPT_COMMAND="${PROMPT_COMMAND:+$PROMPT_COMMAND; } _setAndReloadHistory"

  touch ~/.bash_sessions_disable

The solution for me was storing bash history separately for each vscode project - https://gist.github.com/Srakai/9f3788cdb07259d65d335ff150c38...

Scripts are cool, but imagine ever evolving script that is controlled by arrows and enter

> Why save vim /etc/rc.conf when sudo vim /etc/rc.conf is what I meant 100% of the time?

Good point, but AFAIK all shells search history by recency by default. Whether I search with Fish's up-arrow, or with Atuin's ^r, the first result will be the match I executed most recently. In this case, that means I'd have to type `sudo vim /etc/rc.conf` correctly one time and then that's the first version of the command I'll see next time I look for it. And if it's something I do often enough for it to matter, but still manage to screw it up frequently, I'll turn it into an alias or function.

This is the kind of busywork that feels like an ADHD tarpit to me. No. I absolutely do not need to optimize my shell history, lest I end up with a beautiful, tiny history file that's free from the detritus that would have come from me spending that time actually doing my job.

I do pull very often run things into functions and source them into every shell session. Usually only parameterized things that aren’t trivial to use reverse search for.

The referenced "I don't keep history" philosophy is madness. You won't know what thing would have been useful to keep until you need it $later. Sure, you'll write down some good stuff and maybe alias it.

That's fantastic, do more of that anyway.

Don't pretend you're going to do that for every trick or gotcha you encounter, and don't think you're going to remember that one-off not-gonna-need-it thing you did last week when you find that bug again.

My local history is currently 106022 lines, and that's not even my full synchronized copy, just this machine. It's isolated per-session and organized into ~/.history/YYYY/MM/machine_time_session hierarchy. It has 8325 "git status", 4291 "ll", 2403 "cd .." and 97 "date" entries which don't matter. Literal/complete entries, not including variations like "date +%Y%m%d" which are separate. I can ignore them, either by grepping them out or filtering mentally, but something as benign as "cd .." is INCREDIBLY useful to establish context when I'm spelunking through what I did to debug a thing 2 years ago.

The even better version of both of these variants is to keep everything AND curate out useful stuff. That whole history (4 years locally) is 10MB, and my entire history compressed would be less than a megabyte.

Edit: just realized who posted this, I overlapped with Tod at my first gig in Silicon Valley. Small world!

https://blog.atuin.sh/atuin-desktop-runbooks-that-run/

Im not sure if its entirely what you are asking about though

Shoot me an email if you fancy trying it

Ellie at Atuin dot sh

https://github.com/atuinsh/atuin