Full text search on Fictionmania stories

Looking for stories (or links or ... ) in a certain transformative genre? Looking for a story you can only half-remember? Ask for guidance from your fellow members here.

Full text search on Fictionmania stories

Postby estranhosidade » Tue May 11, 2021 8:03 am

So, one thing that I always found annoying on Fictionmania is that you are forced to use a tag based system, you can't search for stories containing a given word, like "tights" or "forced" or what have you. I end up downloading all stories from the site and creating a database file with it. The file has about 1,5GB and contains 37.000 stories give or taken, it's all divided into different columns split through tabs, if you are on linux, I created a little script that search through the files using FZF. If you are on Windows, I would recommend to search EmEditor which is an amazing easy-to-use software that is able to open these very large files.

Example of search working:
Image

My Script:
Code: Select all
for (( ; ; ))
do
clear
read -p 'Search Term: ' "url"; \
cat Fictionmania.xml | grep -i "$url" | awk -F'\t' '{print $1,$2,$3,$4,$5,$6,$7,$8,$9}' OFS='\t' | awk '{print substr($0,1,50000)}' |awk -F"\t" '!seen[$2]++'  | fzf --preview-window=right:70%:wrap --margin 1,4  --preview "echo {} | awk -F'\t' -vOFS='\n\n' '{print \$1,\$2,\$3,\$4,\$5,\$6,\$7,\$8,\$9}'" --bind 'F2:execute-silent(echo {} | awk -F"\t"  "{print \$1}" | xargs cmd.exe /C start)'
clear
done


Database with stories:
https://archive.org/download/tgbackups/FM-db.xml
Last edited by estranhosidade on Wed Mar 20, 2024 7:21 am, edited 2 times in total.
estranhosidade
Transformation Master
 
Posts: 156
Joined: Fri Jan 08, 2016 4:27 am

Re: Full text search on Fictionmania stories

Postby Adobeman10 » Tue May 11, 2021 8:34 am

Well done, well beyond my knowledge to do-but well done
Adobeman10
Transformation Grand Master
 
Posts: 456
Joined: Sat May 26, 2012 2:16 pm

Re: Full text search on Fictionmania stories

Postby sequoia » Tue May 11, 2021 10:18 pm

This is fantastic. How did you download all the stories without getting flagged as a bot? Do you have a crawling script you could share with us? Send me a PM if you don't want to post it.
sequoia
Member
 
Posts: 8
Joined: Tue Jan 05, 2021 9:03 pm

Re: Full text search on Fictionmania stories

Postby estranhosidade » Wed May 12, 2021 3:39 pm

sequoia wrote:This is fantastic. How did you download all the stories without getting flagged as a bot? Do you have a crawling script you could share with us? Send me a PM if you don't want to post it.

Well, I first got the URLs of all stories and then, since their server is too sensible, I just download them through wayback machine (turns out most of the stories were archived there as well through the years, and wayback machine servers deal better with a lot of data being downloaded without flagging you). After that, I just run the bot every 2 weeks or so, the crawling bot grabs the new stories URLs, check to see how many of them are already in the archive, and then it downloads just the new ones straight out from Fictionmania, and submits the new URLs to wayback machine as well. And since I'm downloading just 30 stories or something, I don't get flagged as a bot nor their servers get overwhelmed.

Here's my code. It's not elegant at all, it's all done on bash, with a little bit of perl, aria2c and other programs, but it does the trick, if I was more skilled, I probably should rewrite that on Python or something.
https://pastebin.com/raw/Tftfss61
Last edited by estranhosidade on Wed May 12, 2021 3:51 pm, edited 1 time in total.
estranhosidade
Transformation Master
 
Posts: 156
Joined: Fri Jan 08, 2016 4:27 am

Re: Full text search on Fictionmania stories

Postby estranhosidade » Wed May 12, 2021 3:43 pm

If anyone is interested I did the same thing with all writing.com stories as well, I have a searchable database with their stories as well. It works more or less the same way:
Image

Code: Select all
for (( ; ; ))
do
clear
read -p 'Search Term: ' "url"; \
cat story-database* | awk -F'\t' '{print $1,$2,$3,$4}' OFS='\t' | awk -F"\t" '!seen[$2]++' | egrep "$url" | fzf --preview-window=right:70%:wrap --margin 1,4  --preview "echo {} | awk -F'\t' -vOFS='\n\n' '{print \$1,\$2,\$3,\$4}'" --bind 'F2:execute-silent(echo {} | awk -F"\t"  "{print \$1}" | xargs cmd.exe /C start),F7:execute-silent(echo {} | awk -F"\t"  "{print \$4}" | sed "s#.*MERDA##" > audio.temp ; balcon.exe -f "audio.temp" -w "audio.wav" -n Salli ;  cmd.exe /C start audio.wav)'
clear
done


(If the code seems weird is because I'm using WSL on Windows, so there's what there's those .exe files in the middle, but it must work on Linux just fine)

Writing database:
https://archive.org/download/stories_data/story-database.xml
estranhosidade
Transformation Master
 
Posts: 156
Joined: Fri Jan 08, 2016 4:27 am

Re: Full text search on Fictionmania stories

Postby Agriculturallaborer » Sun May 16, 2021 6:14 am

Do you have any pointers on how this would be searched on windows?
Agriculturallaborer
Member
 
Posts: 6
Joined: Mon Oct 14, 2019 8:26 pm

Re: Full text search on Fictionmania stories

Postby estranhosidade » Wed May 19, 2021 9:07 pm

Agriculturallaborer wrote:Do you have any pointers on how this would be searched on windows?

Download the software "EmEditor", it's a pretty good software to open these really large text. Here what searching on the database through EmEditor most look like
Image
estranhosidade
Transformation Master
 
Posts: 156
Joined: Fri Jan 08, 2016 4:27 am

Re: Full text search on Fictionmania stories

Postby tfrmer » Thu May 20, 2021 2:41 pm

That's incredible work! Do you have a way to download writing.com stories? I made a bunch years ago, but without paying money I can't read any of what I wrote. I will have to try that soon as there were some m2bbw type stories buried on FM I would love to rediscover.
tfrmer
Member
 
Posts: 56
Joined: Sun May 27, 2012 10:24 am

Re: Full text search on Fictionmania stories

Postby estranhosidade » Thu May 20, 2021 3:39 pm

tfrmer wrote:That's incredible work! Do you have a way to download writing.com stories? I made a bunch years ago, but without paying money I can't read any of what I wrote. I will have to try that soon as there were some m2bbw type stories buried on FM I would love to rediscover.

I created a writing.com backup database with all stories I could find from previous backups and from wayback machine copies as well. It's a little tricky to navigate, basically guide yourself through the URLs inside the index I created instead of links inside from the chapters themselves.
https://docs.google.com/spreadsheets/d/1No4zjBzV81O9G3lNx6Z6ib3idqIdJEH_KCAcVkO0VnE/edit#gid=901689988

Also, that 3GB file I posted before (https://archive.org/download/stories_da ... tabase.xml) contains all writing.com stories as well, you might download that, and it's as searchable as well, you mgiht open it on EmEditor++ as well.
estranhosidade
Transformation Master
 
Posts: 156
Joined: Fri Jan 08, 2016 4:27 am

Re: Full text search on Fictionmania stories

Postby setebos » Sat May 22, 2021 12:59 pm

you have done a great service to the community, truly, thank you
setebos
Member
 
Posts: 22
Joined: Tue Nov 20, 2018 12:15 pm

Next

Return to Looking For ...

Who is online

Users browsing this forum: breif and 9 guests