Follow

help wanted, @ all mastodon admins 

Hello fellow Mastodon admins, could you check your log for mastodon-sidekiq.service for Segmentation faults for ":blurHashForPixels"?

You could typically check by doing the following:

$ journalctl -u mastodon-sidekiq

type a slash (/) and then type Segmentation. if there are any results, please let me know. This may be useful for a bug report.

· · Web · 3 · 10 · 7

re: help wanted, @ all mastodon admins 

@kescher Can't find anything (since 16th of may this year at least).

uname -a: Linux McBeth 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

mastodon version: glitch-soc/mastodon (v3.5.2+glitch).

re: help wanted, @ all mastodon admins 

@Vierkantor wow, 3.5.2. mind telling me which commit hash?

re: help wanted, @ all mastodon admins 

@kescher b91196f4b73fff91997b8077619ae25b6d04a59e

re: help wanted, @ all mastodon admins 

@kescher I should probably update at some point, but between dream and deed stand paper deadlines and practical objections :(

help wanted, @ all mastodon admins 

@kescher Yup, quite a few of these here

The logdumps from these are quite long[0]; I've uploaded the latest one to gist: gist.github.com/7b0585cc1ef9de

[0]:
zorigami# journalctl --unit=mastodon-sidekiq.service --since='2022-09-22 08:00' | awk '/Segmentation/ { p=1 } /mastodon-sidekiq.service: Main process exited, code=dumped/ { p=0 } p { print }' | wc -l
9921

help wanted, @ all mastodon admins 

@ar interesting. Question: Did you compile ruby (with rbenv install) with jemalloc support per the documentation?

help wanted, @ all mastodon admins 

@ar (I know for a fact that it isn't the images themselves causing the segfault, that's why I'm trying to figure out common causes)

help wanted, @ all mastodon admins 

@kescher I'm mostly using the nixos packages/modules for mastodon deployment (+ 2 small patches, one for extending post length, and one for giving moderators full emoji management); It does appear to add jemalloc to LD_PRELOAD, and it does appear in the process maps[0], but I haven't configured it myself one way or another.

[0]:
zorigami# grep libjemalloc /proc/$(pidof sidekiq)/maps
7fa969200000-7fa969209000 r--p 00000000 103:03 1461182871 /nix/store/akwggdprh19p1z96pcgi8mf9qwc2m50q-jemalloc-5.3.0/lib/libjemalloc.so.2
7fa969209000-7fa96929d000 r-xp 00009000 103:03 1461182871 /nix/store/akwggdprh19p1z96pcgi8mf9qwc2m50q-jemalloc-5.3.0/lib/libjemalloc.so.2
7fa96929d000-7fa9692b2000 r--p 0009d000 103:03 1461182871 /nix/store/akwggdprh19p1z96pcgi8mf9qwc2m50q-jemalloc-5.3.0/lib/libjemalloc.so.2
7fa9692b2000-7fa9692b3000 ---p 000b2000 103:03 1461182871 /nix/store/akwggdprh19p1z96pcgi8mf9qwc2m50q-jemalloc-5.3.0/lib/libjemalloc.so.2
7fa9692b3000-7fa9692b9000 r--p 000b2000 103:03 1461182871 /nix/store/akwggdprh19p1z96pcgi8mf9qwc2m50q-jemalloc-5.3.0/lib/libjemalloc.so.2
7fa9692b9000-7fa9692ba000 rw-p 000b8000 103:03 1461182871 /nix/store/akwggdprh19p1z96pcgi8mf9qwc2m50q-jemalloc-5.3.0/lib/libjemalloc.so.2

help wanted, @ all mastodon admins 

@kescher That said, the ruby interpreter used by sidekiq does not appear to have jemalloc enabled:
zorigami# /nix/store/7x8g66l4n49d3nwjccdqmzyzbkvv82jx-ruby-3.0.4/bin/ruby -r rbconfig -e "puts RbConfig::CONFIG['LIBS']"
-lm

help wanted, @ all mastodon admins 

@ar that doesn't mean much, since mine is definitely compiled with jemalloc, but also doesn't show the option.

help wanted, @ all mastodon admins 

@ar Hmmm... What happens if you don't preload it (remove the LD_PRELOAD for mastodon-sidekiq)? Does it disappear from the maps?

help wanted, @ all mastodon admins 

@ar (if yes, try that and see if that solves the issue - that's what I am currently trying as well)

help wanted, @ all mastodon admins 

@kescher set the LD_PRELOAD to empty value and jemalloc disappeared from maps.

help wanted, @ all mastodon admins 

@kescher And gonna see in a few hours if the segfaults still happen

help wanted, @ all mastodon admins 

@ar I'd try monitoring it over days lol

help wanted, @ all mastodon admins 

@kescher I've had it happen around 10-15 times per day since September 19, so I would assume a few hours would suffice.

help wanted, @ all mastodon admins 

@ar wow, 10 to 15 times? It happens around once, maybe twice a day here.

help wanted, @ all mastodon admins 

@kescher And now that I look at it, it only started happening on September 19, at least in my instance

help wanted, @ all mastodon admins 

@ar that is really weird.

help wanted, @ all mastodon admins 

@kescher Well, so far, no new segfaults here :blobcatjoy:

help wanted, @ all mastodon admins 

@ar wacky.

help wanted, @ all mastodon admins 

@kescher Still no new segfaults. I guess I'd say that removal of jemalloc fixed things.

help wanted, @ all mastodon admins 

@ar Yeah. I suppose this will go into a bug report.

help wanted, @ all mastodon admins 

@ar Thanks for sharing this, btw!

help wanted, @ all mastodon admins 

@kescher n/p

Also, the bit about segfaults appearing only after September 19 was wrong - I forgot I have quite aggressive log rotation configured

Show newer

help wanted, @ all mastodon admins 

@kescher never restarted for me

Sign in to participate in the conversation
CatCatNya~

We are CatCatNya~, a left-wing instance by cats, for cats (and more!).