>>16524 >We will sponsor work visas to Tokyo or San Francisco if necessary! Eww! The job offer is tempting, the prospect of foreign multi-racial team is not, I guess it has to do with the alien nature of the money funding the enterprise.

>>16524 Running from source on Manjaro Linux. I updated from v596 to 598 and MPV cannot be found. I refreshed the Venv using the (N)ew and also (T)est MPV options, but no success so far. By the way, this Manjaro install is a fresh one and it is possible some system libraries are not longer present.

>>16529 Same anon here. I fixed it. Re-reading my post >>16529 about missing system libraries I had a light-bulb moment. In the previous Manjaro install, the MPV player was installed and checking its libraries I found it has a MPV.SO one. So, I installed it and now the MPV viewer in Hydrus is working as expected.

>>16522 I don't think that's what I'm looking for. I'll be honest, I rarely mess with subscriptions but from what little I use it, it'd still be pointless. Hyrdus picks up each image and image set based on url so if I tell hydrus to grab the first 5 urls I'd get something like this on pixiv >url 1 [single image] >url 2 [single image] >url 3 [image set of 10 images] >url 4 [image set of 10 images] >url 5 [image set of 100 images] Hyrus will look at that as 5 urls but each url containing multiple images. So if there's 10 urls like [image set of 100 images], that's already pushing 1,000 images, which is what I've been seeing on that site. I don't think there's anything you can do about this however, I feel like I'm going off topic from what originally I want. I just want hydrus to not fetch and display 10k images all at once when I go to check on my gallery and url downloads. I still want to download them just not waste resource trying to display them all at once. A display limit like with the search page but for gallery downloader is what I want. That's why I suggested an infinite scroll type function. You scroll down to a point then Hydrus would fetch another batch where you could set how many it can fetch. I used to hate infinite scrolling on most sites but for Hydrus it makes the most sense.

>>16518 I tried about 10 different libmpv dlls, but none of them resolved the issue. Some .dlls showed grid lines, others just gave an error in hydrus. example:20240304 v597, win32, frozen ValueError ('Invalid value for mpv parameter', -4, (<MpvHandle object at 0x0000019619A63E50>, <mpv.LP_MpvNode object at 0x000001961B41AA50>, <mpv.LP_MpvNode object at 0x000001961B41A9D0>)) File "hydrus\client\gui\canvas\ClientGUICanvas.py", line 1637, in SetMedia File "hydrus\client\gui\canvas\ClientGUICanvas.py", line 1294, in SetMedia File "hydrus\client\gui\canvas\ClientGUICanvasMedia.py", line 2214, in SetMedia File "hydrus\client\gui\canvas\ClientGUICanvasMedia.py", line 1587, in _MakeMediaWindow File "hydrus\client\gui\ClientGUI.py", line 7518, in GetMPVWidget File "hydrus\client\gui\canvas\ClientGUIMPV.py", line 293, in init File "mpv.py", line 1338, in loadfile File "mpv.py", line 1229, in command File "mpv.py", line 142, in raise_for_ec I need to correct myself, both animated gifs and webm ARE affected.

I doubt this is supposed to happen.

For some reason this file is playing really really fast on Hydrus 596. It didn't always do that.

Is redgifs download fucked? They seem to have changed something and when I try to open the link in browser and view source and visit what appears to be the file link I get blocked. >>16538 I've noticed this on a few files as well, including a few webms. Can fish some out if needed

>>16535 Wanted to report this visual bug too, found on all three tabs of a new 'duplicates processing' page. Also Hydrus nas noticed me that 'meta:contentious content' was blacklisted for pending tags to PTR it seems. Just to understand it right, it does mean that this tag doesn't get uploaded to PTR anymore i guess? What is the reason? Is it because 'contentious content' has no real definition and is defined different by everyone? What about the 778k files in PTR that have that tag already. Will they keep it or lose it in the future?

>>16538 https://onlinegiftools.com/change-gif-speed Put in the 'fast' gif and check the frame delays. As you can see, the info says: Total frames: 64 Input GIF duration: 0.86s All Input GIF Delays: Frame 1: 70ms Frames 2-22, 24-37, 39-63: 10ms Frames 23, 38: 50ms Frame 64: 90ms Afaik: that means that the 'fast' playback that you are complaining about, seems to be the real playback speed. Most of the frames play at 10ms. They are set up like that. The question would be, what is the intended speed of the creator and what tools were used to create the gif with those custom delays? I think intended would be 100ms instead of 10ms for the frames that have this delay (Frames 2-22, 24-37, 39-63), looks correct. Some mistake must have happened creating the gif. Players/browsers (or websites?) that play it slower, have a minimum frame delay for each frame (like 100ms) and/or they cannot play custom delays for individual frames and therefore play it wrong. Cannot pinpoint which is it in chrome for example (also not how it was in older Hydrus versions). Seems to be something around 95ms weirdly enough. Maybe thats the average of all individual frame delays. Now you will find probably more gifs that play fast. My imageviewer 'irfanview' plays them also fast, therefore correctly i guess. Not sure why .webms should change though like the other guy suggested, but i cannot comment on that. You can save the new gif but of course the hash changes and also filesize ever so slightly. But better wait till Hydev answers, he should know better what to do.

>>16541 >Maybe thats the average of all individual frame delays. I mean average if the 10ms times would be discarded because they are too fast and they would be exchanged for 100ms frames or something like that.

>>16538 >>16539 >>16541 Okay, I tried to attach some webms that play fast in the native viewer yet fine in external mpv but 8chan says it's a format not supported by the site. Bizzarely enough, `file` says that at least the one I tested is an EBML file with the creator \004webm. I've never heard of that, apparently it has something to do with .mkv? I'm guessing whoever is encoding these is fucking them up somehow. The artist sky_necko on e621 has a high rate of fast webms, about 3/4 of the ones I have saved play fast. Here's an example: https://e621.net/posts/5114166 Obviously furshit so don't click if you're triggered by that.

Random suggestion: search history that persists through restarts.

>>16504 >Ah understood. In case anybody worries about data privacy or security: does that mean that if you only use Hydrus on an external drive and use that option, files you DnD from Hydrus into a folder on the same external drive, it first gets copied to C: then renamed, copied back and then deleted from C: with a potential recovery possible? For temp dir privacy, I think the bulletpoints are: - on boot, hydrus creates a new directory in your temp dir and does all its work in there. SQLite spools its larger transaction stuff to there too - on a clean program exit, the directory is deleted - on a non-clean program exit (i.e. a crash), the directory is not deleted and it is now up to your OS (or you) to eventually get to it - every import file travels through this temp dir. it is deleted moments after the import is finished - every small drag and drop with that compatibility mode exports the files to a new sub-dir inside our temp dir. the change I made last week is to delete this subdir after six hours instead of waiting until program exit. The DnD mode only does the temp dir thing if the DnD is <=50 files and <250MB I think, to keep things snappy if you are moving heavy stuff just from one tab to another. If you cannot trust your temp dir, you can choose a different location using the '--temp_dir' launch switch: https://hydrusnetwork.github.io/hydrus/launch_arguments.html#--temp_dir_temp_dir I set the new temp location in your environment path very early in boot, so I am pretty sure that everything, including SQLite, will use the new location. You can see your current temp dir in help->about btw, on Windows it'll probably be C:\Users\YOU\AppData\Local\Temp. Anything that hydrus used is prefixed 'hydrus'. >>16505 Thanks, that's interesting. I regret the tangle most of this UI became, with the crazy nested menus. I have many plans to improve it, but there's plenty of behind the scenes work to do first. Fingers crossed, duplicate auto-resolution will be a bit of, and the larger beginning of, a relief. >>16506 >opening files in a new duplicate filter page Ah, thanks, I had forgotten you could do that! I guess that new dupe filter page inherits the previous file context. This sounds like a great place to have a hyper specific checkbox in the options--I'll see what I can do. >>16512 >>16515 >>16516 >>16517 Thanks. I think we can say that the current system is very good at detecting duplicates and sometimes iffy on detecting alternates. Most of my time has been on duplicate handling so far, so I'm mostly happy with this. As for the duplicate ponies, and future alternate handling and costume/WIP detection, I suspect the solution here is going to be to take multiple perceptual hash snapshots of files, in different subsections. For instance, for each of those three pairs, the differences are almost entirely in the top or bottom half of the image, so a phash system that similar data just for the bottom and top halfs of the files would detect these at distance 0 or 2. I understand there are some algorithms that handle this question even more intelligently by drawing bounding boxes around the most interesting content in an image, and these help detect crops, too. My system is totally fine with having multiple phashes per file, so this won't be a super difficult thing to engineer, but we'll need to do lots of testing to make sure we don't overwhelm ourselves with false positives through bad phash section choice. There's more work to do, but I think we'll get there.

>>16520 >>16521 >>16522 Yeah, I broadly agree. I made the decision not to do any booru-style 'pagination' when I first started hydrus, but as we've pushed the boundaries and come to regularly hit 20-30k file pages for certain queries, we hit lag city. I've optimised over and over, but I finally surrendered on the 'selection tags' problem last week by applying the 4096 default limit. The thumbnail grid is unfortunately one of the worst-coded areas of hydrus (and that is saying something!), and is in need of a complete overhaul. It is all brittle ugly hardcode right now, but me and a guy think it may be possible to convert it to entirely Qt widgets, even up to millions of files on a page. This won't solve this 'streaming'/pagination issue, but it will move us to flexible Qt code which will allow the implementation of more dynamic loading far more easily than my ten year old bullshit. Watch this space, but I can't promise anything any time soon. This is one of those back-burner cleanup jobs where I hope to suddenly one day get to it and move us to something nicer without actually changing any of the front-facing features at all, and then I'll be in a position to think about new presentation options, including stuff like a list view instead of thumbnails. As for this >>16533 , yep, unfortunately pixiv is just a bit crazy in ten different ways. I built the downloader, broadly speaking, for the shape of a normal booru, and if you want fine control over a pixiv import stream I think you have to ride the 'pause/play files' button. Either drive the import manually, and set 'skip' on stuff you don't want, or operate on a 'whitelist' where you drag and drop good pixiv URLs onto the client manually, or just surrender to getting spammed with sixty messy variations of a random CG. I fucking hate 'multiple files per post URL' as a hosting concept, but there we go. I can't provide 'scroll down to keep downloading', but surfing the 'pause/play files' button works well if you want to get a broader preview of what an artist has to offer. Also, the bigger artists are all on the normal boorus, where the spammy content is culled, so that's another option--just look their most popular files up on saucenao and find them on a saner site. >>16529 >>16530 Hell yeah, well done--for future users who run from source, what could I add to the 'how to get mpv mate' section of the 'running from source' help? (https://hydrusnetwork.github.io/hydrus/running_from_source.html#built_programs here, Linux tab) I know very little about Linux, but if I said 'You can try checking your package manager for mpv too--if it says it bundles libmpv.so, you can just install it and that will probably work too.', would that sort of thing be correct and sufficient? I know that some builds of the mpv video player do not come with libmpv, but some do. I guess it is a static vs dynamic dll build thing. >>16534 Damn. Thank you very much for testing. I feel like a shit now, since I think I'm going to move forward with the upgrade. We want the new tech for the wider userbase because the new mpv loads smoother and fixes some gif problems, and if a handful of users on more unusual OSes get some black bars, that's at least better than outright import fail errors. I'm going to think a bit more, but I suspect I will fold the new mpv into the normal release and have special instructions for users with problems on either replacing their mpv manually (would be annoying and have to do every week), or moving to running from source (only have to do it once, very flexible to local circumstances). I think I will do it hesitantly--I'll roll it into v599, and if many many other users have black bars or other trouble in the wider test, then I'll roll back in v600. I have to guess it is Windows Server causing the trouble though. I know one guy on Win 10, which was my main worry, who had no problems.

>>16535 >>16540 Aiiiieeeeee! Thank you for this report. I had to fix a bugged multi-column list id last week, and I bet it screwed with your column widths in the new placeholder 'auto-resolution' tab. Please hit that tab, right-click the tab header and say 'reset column widths for "review duplicates auto-resolution rules"' (it should say auto-resolution, not anything to do with export folders. If it says export folders, I think you are on v597, please update to v598). You may need to restart the client to get it to lay out again and fix your fucked panel width. If you cannot see the 'auto-resolution' tab, please turn on help->advanced mode and restart the client. >>16540 For PTR stuff: Yeah, the PTR tag filter just stops your client from uploading that in future. No worries, and you don't have to do anything, just keep on parsing like normal, it is all handled for you. There's a similar tool the jannies use that deletes it from existing content en masse, so I expect it will go from those 778k files in the near future. Booru-specific stuff like 'do not post' and 'tag request' and 'very high res' makes sense on the site but not for our purposes on the PTR, so it is simpler to just block it all than try to alter our parsers to not grab it. I didn't talk to our guys about this recent 'contentious content' block specifically, but I think I'm right in saying that tag specifically refers to booru content policy, to differentiate spicy content that you need to be logged in or whatever to see? It is a tool for booru-side filtering in one way or another, rather per se than an actual descriptor to be used neutrally. If you are interested in seeing the whole current tag filter, it is under services->review services->remote->tag repos->PTR->network sync->tag filter. >>16538 >>16539 >>16548 Thank you! 12fps is my fallback speed for when I cannot figure out a gif duration/framerate, so I bet this guy is parsing wrong. I will look into it. Broken files are always welcome; even if I cannot fix them, they are useful to have around. If a site won't let you post, or the CDN modifies the file and fixes it mid-transit, then you can always zip them up and catbox them to me. And yeah, my native viewer obeys my frame timings, but mpv will ignore what I suggest and do whatever it thinks is best, which is almost universally better than what I figure out. That e621 file looks a little crazy. It has a duration of ~17 seconds in hydrus and Firefox and MPC-HC for me, and it looks like mpv is rendering at the same speed. Hydrus reckons it has 1,691 frames for 100fps, but I don't know if it is lying about the actual number of frames since when I try to tell hydrus's mpv to go forward one frame, it jumps like 6-8. What do you see? When you get a nicely rounded but weird number like '100fps', it is not usual that someone set up a conversion wrong. If we are entering the magical world of mkv muxing, then yeah I expect some human or automatic ripper screwed up a conversion here and the header on the webm is straight-up wrong. >>16549 Man, I really need to figure out a proper search undo. I hate removing a search predicate and then realising I removed the wrong thing. I totally agree about getting a history on search pages, and then attaching it to the session so it persists.

>>16552 >What do you see? I'm getting the same 16.9 seconds, 1691 frames, 100 fps in the bottom bar when it is selected in the gallery. The native viewer plays it all in about 2 seconds and holds the last frame until it 'repeats' at 16.9 seconds. Opening it in mpv (externally) I see an estimated 12.0482 fps. ffmpeg says it's a "matroska,webm" that's 16.91 seconds long but I can't get it to tell me how many frames there are. If it at all helps, the original file is an ugoira. Here's a direct link: https://files.catbox.moe/8ro11c.zip Hydrus shows that ugoria as having 203 frames and is estimated to be 25.4 seconds long, and plays at a more normal speed. I'm on Linux, version 595 if it matters.

Random minor UI nitpick: when you go to import a downloader, the menu only says to drag and drop them on Lain without indicating that you can open a file picker instead by clicking on her or by clicking the clipboard to paste a copied image (though the clipboard at least has a mouseover indicating this). Also, maybe the clipboard could also detect if a path to a downloader is copied instead of the bitmap itself?

>>16553 Yeah it looks like a busted file header. FFMPEG reports 100fps, but I forced my 'distrust fps, read frames manually' thing, and it comes back as 203 frames for 12fps. I've written a thing in my file parser to say 'distrust 100fps mate', so we should catch these in future. I'll schedule a round of metadata for all small video with 100fps on update. The bra gif here >>16538 seems simply not fully readable by PIL, which I use for gif durations. The first frame seems to be 70ms, but those after are 1ms, which is often shorthand for a null result. I expect the frame headers are busted or in an unusual int format or something. Typically in these cases a future version of Pillow will suddenly start reading them correctly. >>16554 Thanks, I will clean this up!

Same anon of >>16529 and >>16530 posts and running v598 from source on Manjaro. Hydrus was randomly hanging the whole computer when clicking the small viewer on the lower left corner to pause a video, or, double-clicking a thumbnail to maximize a video, forcing me to hard reset the machine. After some troubleshooting I discovered that the default viewer was set as Qt Media Player instead of the old and dear MPV. So, I reverted it to MPV and the crashes are gone. >>16551 >what could I add to the 'how to get mpv mate' section of the 'running from source' help? Perhaps, just to be on the safe side, the Help Source Page may explain that if the Distro's Package Manager does not list any libmpv library, then it might be worth to try installing the ubiquitous MPV Player, which may have it.

>>16503 >Reverse sorting problem. Playing around it needs to not do any resorting at all from what the HTML provides. The HTML page is in the correct order too. Pools can have non-sequential post IDs which usually happens when the posts are re-uploaded at better quality or the original uploaded didn't do them in order. Pools allow you to adjust the ordering by manually listing the post IDs in the desired order so string sorting the HTML will break the actual sequence. Everything seems to be perfectly fine with the parser and every file is in the correct order under the "file log". It appears that the problem is near the end of the chain when dropping files into the thumbnails page.

>>16552 >reset column widths cheers, this fixed it

What was the trick to downloading 8moe threads again? I sent my session cookie to Hydrus, but attempting to download a thread still hits the splash page.

>>16550 >I understand there are some algorithms that handle this question even more intelligently by drawing bounding boxes around the most interesting content in an image, and these help detect crops, too. My system is totally fine with having multiple phashes per file, so this won't be a super difficult thing to engineer, but we'll need to do lots of testing to make sure we don't overwhelm ourselves with false positives through bad phash section choice. I used the tool AllDup a little bit and its very mighty. It can find similar images with different comparison methods (aHash, pHash etc), finds you files from a ton of different properties etc. Might be a bit overwhelming first, but can't recommend it enough. Check out the documentation, it may help you for your plans with hydrus. https://www.alldup.de/alldup_hilfe/alldup.php Scroll down for some description in english, what it can do. https://www.alldup.de/alldup_hilfe/index.php (Documentation) https://www.alldup.de/alldup_hilfe/search_similar_pictures.php Scroll down for nice tables (recognition rate test) and so on. Maybe it's some help and give you some ideas? 'AllDup is Freeware and can be used free of charge by private users and companies.' Since it finds you all kinds of files, people can use it for finding duplicates before importing into hydrus too.

>>16552 >That e621 file looks a little crazy. It has a duration of ~17 seconds in hydrus and Firefox and MPC-HC for me, and it looks like mpv is rendering at the same speed. Hydrus reckons it has 1,691 frames for 100fps, but I don't know if it is lying about the actual number of frames since when I try to tell hydrus's mpv to go forward one frame, it jumps like 6-8. What do you see? It's the same for me. Checking different webms with MediaInfo i can say, that this one is a special one. This one gives: Format : WebM Format version : Version 2 File size : 5.88 MiB Duration : 16 s 910 ms Overall bit rate : 2 917 kb/s Writing application : whammy Writing library : whammy Video ID : 1 Format : VP8 Codec ID : V_VP8 Bit rate : 2 795 kb/s Width : 500 pixels Height : 500 pixels Display aspect ratio : 1.000 Frame rate mode : Variable Compression mode : Lossy Default : Yes Forced : No Whammy (google: Whammy: A Real Time Javascript WebM Encoder) is unusual. I checked a bunch of files, and all of them are written with Lav. The frame rate mode here is 'variable' which could also play a role in this, but from my webm files, very few were also 'variable' but played correctly in the native player and also mpv. So i guess whammy is the reason, plus potentially the variable frame rate mode in combination.

>>16570 Maybe this helps: >>16408

Hi, is there a way to compare subscription lists to see what subscriptions are missing artists that others have?

>>16574 what I do is copy the queries of both subs I wanna check, paste them into 2 separate files, then I just go to the command line and diff sub-1.txt sub-2.txt

>>16573 I have the correct user agent now, but now Hydrus Companion seems to be failing get any cookies from 8moe.

I had an ok week. I fixed some UI issues, improved how file timestamps display, and optimised database vacuum maintenance. The release should be as normal tomorrow.

The e621 parser component appears to have broken, no longer returns results.

>>16579 I fixed it (for myself) by going to downloader components > parsers > e621 gallery parser > content parsers tab > file page urls content parser (that produces downloadable/pursuable urls) > edit formula > the first search descendants, its key=value changed from class=post-preview to class=thumbnail

https://www.youtube.com/watch?v=UtG11pmBe3U windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v599/Hydrus.Network.599.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v599/Hydrus.Network.599.-.Windows.-.Installer.exe macOS app: https://github.com/hydrusnetwork/hydrus/releases/download/v599/Hydrus.Network.599.-.macOS.-.App.dmg linux tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v599/Hydrus.Network.599.-.Linux.-.Executable.tar.zst I had an ok week. I fixed some bugs, improved some quality of life, and overhauled vacuum maintenance. Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html highlights If you got a superwide duplicates filter page sidebar last week, it should fix itself today. It was an UI bug from a couple weeks ago that wasn't fully cleaned up. The e621 downloader stopped finding files in the past week, but I have fixed it. You don't have to do anything. I understand they may still be making changes on their end, so let me know if anything new breaks. When you see file timestamps in the media viewer's top hover window or the file right-click menu, their tooltips are now the inverse of your 'always show ISO timestamps' setting. So, if it says 'modified: 2 years ago', the tooltip will say 'modified: 2022-11-20 14:23:39', and vice versa! Advanced users only: The database 'vacuum' maintenance task, which is essentially a database defrag, now runs significantly faster (in my tests, maybe 10 times faster, anything from 30-170MB/s, but I suspect super big databases will run ~10MB/s) and no longer needs to use your temp dir. I only recommend running vacuum every, say, five years for a few percent performance improvement, but if you have been waiting to clean up or truncate a huge and tangled client.mappings.db, it should be easier to find the space now. I have also added a summary popup that reports how much space the vacuum saved and how fast it worked. I'd be interested in knowing what speeds you see. next week I have nothing exciting prepared for v600. Just some more general work while I continue to chip away at duplicates auto-resolution in the background. >>16579 Thanks, should be fixed in this release.

>>16582 >I have nothing exciting prepared for v600. The question is, what have you prepared for v666? Will we be able to play doom in hydrus?

>>16583 >v666

>>16559 Same anon and an update. It looks like the computer hanging has nothing to do with Hydrus as I already experienced 3 random crashes when launching a video with VLC. So the culprits look like the Qt6 used by Manjaro, the video compositor X11 (i really doubt it), and the most likely guilty party: the open source video driver which is failing to manage properly the video pagination, I mean, it won't release memory by around 500MB, in other words, there is something wrong with the destructors.