Ngposts obfuscation doesn't seem to work

[Oleekae]

not verified by myself as I don't have an account with that provider but I copy and paste the original post from the user who verified it.

"I noticed something
Ngposts obfuscation doesn't seem to work
I have obfuscation set to article
But whatever i upload, well it doesn't show up on public indexers like nzbindex and nzbking but Easynews search is able to find it
Even if obfuscation is set to article, the upload get found and indexed by Easynews search"

Other users have stated the same...

Please if this is how it looks like then this needs a fixed asap!!

[Oleekae]

knock knock... is anyone here?

[mbruel]

I don't think this is possible... maybe Easynews did some tricks to recover files posted by its users (same ip) but I don't see how they could recover the name of the files, especially if it's header protected archives...

[Tensai75]

This is definitely not a problem with ngPost.
The body of the message must contain the yenc header and the yenc header must contain the (obfuscated) filename of the rar file. Otherwise, there is no way to decode yenc chunks back into a file.
Furthermore, if the rar file is not password protected, it is usually sufficient to load only the first message of the rar file and decode it with yenc to get the rar headers and see the filenames inside the rar file.
So if an indexer not only indexes the header information of the messages, but also parses the yenc header information in the body of the message (which easynews most likely does), it is still possible to index the upload based on the filename in the yenc header (or even the filename(s) within the rar file) even if the header information (FROM and SUBJECT) is completely obfuscated.
You just have to assume that any message with the same filename in the yenc header (or filename(s) within the rar file) posted within a certain time period belongs to the same upload.

So the obfuscation is not broken. It is simply not possible to obfuscate an upload completely if it is yenc encoded and the indexer goes the extra mile to parse the body of the message (and yenc-decode some of them) as well.

[Tensai75]

Made some tests with easynews and files without file extension are not indexed.

So ideal obfuscation would be like this:

1. Pack the file(s) you want to upload into a splitted rar file (the more rar parts the better)
2. Generate the par2 files
3. Give every rar/par2 file a random filename without file extension
4. Upload each file in a seperate usenet group (GROUP_POLICY = EACH_FILE) while using article obfuscation

In this case no file will be indexed by easynews.
However, the generated NZB file must state the original filenames in the file subject. In this case NZBGet and also SABnzbd will give precedence of the filename in the NZB file over the filename in the yenc header and the downloaded files will have the correct (original) filename and can then be repair (if needed) and extracted.

So to improve obfuscation, ngPost would need to do the filename obfuscation according to step 3 and when using filename obfuscation, the filename in the file subject in the generetade NZB file must be the original filename.

I would also suggest that filename obfuscation should not obfuscate also the filenames within the rar. This should better be a seperate option.

[Tensai75]

Suggestion to implement this the most easy way:

If _obfuscateFileName is true simply write the md5 hash (or any oder hash) of the filename instead of the filename iself:

ngPost/src/nntp/NntpArticle.cpp

Line 66 in 7f4762b

<< " size=" << _nntpFile->fileSize() << " name=" << _nntpFile->fileName() << Nntp::ENDLINE

If _obfuscateFileName is true simply write the md5 hash (or any oder hash but same as above) of the filename instead of the filename itself (this is to obfuscate file name in the article subject if "article obfuscation" is deactivated):

ngPost/src/nntp/NntpFile.h

Lines 99 to 102 in 7f4762b

	QString NntpFile::nameWithQuotes() const {
	return QString("[%1/%2] \"%3\"").arg(_num, _padding, 10, QChar('0')).arg(
	_nbFiles).arg(_file.fileName());
	}

and

delete the following functions (because changing the filenames is not actually required):

ngPost/src/PostingJob.cpp

Lines 978 to 997 in 7f4762b

	if (_obfuscateFileName)
	{
	_files.clear();
	for (const QFileInfo &fileInfo : _originalFiles)
	{
	QString obfuscatedName = QString("%1/%2").arg(fileInfo.absolutePath(), _ngPost->randomPass(_ngPost->_lengthName)),
	fileName = fileInfo.absoluteFilePath();
	if (QFile::rename(fileName, obfuscatedName))
	{
	_obfuscatedFileNames.insert(obfuscatedName, fileName);
	_files << QFileInfo(obfuscatedName);
	}
	else
	{
	_files << fileInfo;
	_error(tr("Couldn't rename file %1").arg(fileName));
	}
	}
	}

ngPost/src/PostingJob.cpp

Lines 978 to 997 in 7f4762b

	if (_obfuscateFileName)
	{
	_files.clear();
	for (const QFileInfo &fileInfo : _originalFiles)
	{
	QString obfuscatedName = QString("%1/%2").arg(fileInfo.absolutePath(), _ngPost->randomPass(_ngPost->_lengthName)),
	fileName = fileInfo.absoluteFilePath();
	if (QFile::rename(fileName, obfuscatedName))
	{
	_obfuscatedFileNames.insert(obfuscatedName, fileName);
	_files << QFileInfo(obfuscatedName);
	}
	else
	{
	_files << fileInfo;
	_error(tr("Couldn't rename file %1").arg(fileName));
	}
	}
	}

Don't think the remainder of the code need much other changes except that you need to pass the _obfuscateFileName bool to the NntpArticle and NntpFile::nameWithQuotes function.

[nixxou]

Ok, so it's kind of weird but i was working on my own nttp poster tool and while looking about how does ngpost that i'm used to, i stumble on this problem before even seen there was an issue post on the github.

I just want to add something :
Using the md5 hash of the filename is a bad idea because there is no need that all segments use the same fake filename in the yenc header. You can just use a different random string for each segment, nzbget will rename using the file subject in the nzb.
That will make obfuscation far more effective.

Also, i though about some additional ideas to improve obfuscation :

• Randomize the order of file post
• Make a memory stack where a percentage of the YEncFilePart are stored (not necessary all if you don't want to allocate too much memory), while posting, post elements of that stack in random order.

[Tensai75]

Using the md5 hash of the filename is a bad idea because there is no need that all segments use the same fake filename in the yenc header. You can just use a different random string for each segment, nzbget will rename using the file subject in the nzb.
That will make obfuscation far more effective.

Agree, this will strongly increase the obfuscation. But what if you don't use NZBGET for downloading but another tool who relays on the yenc header filename for yenc decoding, instead of the filenames in the NZB file? They will probably not be able to decode the file(s). By keeping the filenames constant for each file you will at least get the decoded files back with obfuscated names. And by guessing that the smallest file will probably be the .par2 file and renaming it to XXX.par2 you can then restore the original filenames.
When changing the yenc filename for each segment you are bound to those download tools who can handle it.
Depending on the intended use case for the obfuscation, this is might be something to consider.

• Randomize the order of file post

Well, the articles still look "similar" and the yenc header still contains the part number. So if such articles are posted within a certain period of time, even in random order, you can still assume that they belong together and try to decode them.
However, if you actually randomly change the file name in the yenc header of each article, I would suggest posting all the files in parallel in the same group, i.e. all part 1, then all part 2, etc.
That way it would be much harder to guess which part 1 belongs to which part 2, especially when uploading split rar files with identical file size of the rar parts. And the more rar parts you have, the harder it gets.
Just an idea....

[nixxou]

Agree, this will strongly increase the obfuscation. But what if you don't use NZBGET for downloading but another tool who relays on the yenc header filename for yenc decoding, instead of the filenames in the NZB file?

I suppose it's quite popular, because i stample on this comparing how i do things vs how does a nzb file that i got on an semi-private indexer and on this file, each segment used a different name inside the yenc.

I also though of something but that involve a lot of preprocessing, ram or disk write : precalculate yenc part for the whole archives, and make "fake files" when you totaly reorder the order of the yenc part (like for a file, you use the first element of the 001, the second of the 004, ...)

[nixxou]

Btw, it's maybe out of scope of this issue, but i would be interested if you can give me an opinion, that my current take on obfuscation. And maybe that can give you some idea for your own project.

My nzb seems to works fine with both Nzbget and sabnzb.

I use splited 7z over multipart rar, because while rar are easier to blend in, i think that splited 7z header can only be read from the 001 archive and when i don't use password, i actually write garbage on few ko of the 7z.001, so if you want to know what are the file in the archives, you will need a repair.

And without the nzb, getting enought pieces together to trigger a repair is not easy, because all yenc part are pre-generated and send in a semi-randomized order.

big_buck_bunny.zip

[mbruel]

I'm not in a state to read all that these days (complicated in my personal life). I thought I was bypassing yenc encoding when doing article obfuscation. if it is not the case it is easy to add. I could even do it for the next release.
the thing is, article obfuscation could and maybe should be taken as spam from the providers as it is kind of.. and they can choose to reject content they can not index. and even ban users that does it. I heard some have done it...

@Tensai75 could you confirm the third point of #175 I'm not entirely sure and don't know how to include you in the conversation directly there... cheers mate

[mbruel]

group alt.binaries.superman 
211 1541927410 157 1541927566 alt.binaries.superman

head <e6c423c782434ea88d9662fa3e50451a@ngPost>
221 0 <e6c423c782434ea88d9662fa3e50451a@ngPost>
From: [email protected]
Newsgroups: alt.binaries.superman
Subject: e6c423c782434ea88d9662fa3e50451a
Message-Id: <e6c423c782434ea88d9662fa3e50451a@ngPost>
Lines: 18
Path: not-for-mail
Date: Mon, 10 Jul 2023 06:19:02 +0000
Nntp-Posting-Date: Mon, 10 Jul 2023 06:19:02 +0000
Organization: theCubeNet - www.thecubenet.com
X-Complaints-To: [email protected]
X-Received-Bytes: 2516
.



body <e6c423c782434ea88d9662fa3e50451a@ngPost>
222 0 <e6c423c782434ea88d9662fa3e50451a@ngPost>
=ybegin part=1 total=1 line=128 size=1812 name=84fGOJF9K9Tb3PwvkJmZlwab1UJitobri.par2
=ypart begin=1 end=1812

Indeed the yenc name is not obfuscated.... well easy to bypass it ;)
but as I said, this could be considered as spam...
well if people are interested, maybe I can workout a solution to be completely invisible ;)

a hint, it's around here that something should be done ;)

[loungebob]

Matthieu, can this be implemented and released (as a Windows compile please :-))

I'd even pay for the completely invisible variant.

Maybe as a distraction from your personal life? I know, self-serving sob.

[heartdeadz]

i would absolutely love an update for this too! :D You are a champ for making this awesome GUI tool on windows, mruel! i would not be doing what i'm doing without it!

when that's said, i do hope it gets updated at some point!

[loungebob]

@mbruel you alive? I have a list of bugs that I would appreciated you'd have a look at. One of them is kind of big, showing the tab green as in completed upload but the main rar file was not uploaded at all (and also shows like that in the tab detail view.

[mbruel]

@Abu3safeer you can't share builds without the code! I'm deleting your post
Do a PR, and we'll discuss about it then I'll integrate it. Otherwise you do it on your on fork. But the code must stay opensource!
(that's for safety reasons too ;))

[Abu3safeer]

You are correct @mbruel , that was a mistake from my side, the builds I shared was from https://github.com/Tr4il/ngPost/
I have sent you an email explaining everything, if you can implement Tr4il/ngPost fix, it should solve easynews and nzbindex methods to dectypt the obfuscation of the Article Header, I don't know much about c++ so I have sent you an email.

[mbruel]

I'm working on a new release for my community. This might be included or at least the fix for alpine and my own obfuscation solution

[Tr4il]

Funny to see my ngPost fork being shared here. I'm not proficient in this stuff and I only added in the patches that @xompage had made, all credit goes to them. Glad to hear you're back working on ngPost @mbruel!

[Abu3safeer]

Easynews did play their game, for that the patch is a necessity, and the returning of @mbruel to work on new version is amazing news.

[heartdeadz]

i would love it if you could make it so the checkbox: File Name Obfuscation keeps the extension of the file.
so instead of auijhsdjkasdnbas <--- it would be auijhsdjkasdnbas.mkv , or even auijhsdjkasdnbas.mp4 and such.

i've noticed that allot of people uses nzbget, and if they don't enable deobfuscation manually, the files do come down super scrambled, and the extension is gone too. which makes them stress out :P

Edit: even had a few people with sabnzbd not being able to deobfuscate the file, which also makes it 100% useless for them.

[loungebob]

@mbruel any progress/updates? I got issues...

[demanuel]

Indeed the yenc name is not obfuscated.... well easy to bypass it ;) but as I said, this could be considered as spam... well if people are interested, maybe I can workout a solution to be completely invisible ;)

Just chiming in. The news provider usually don't parse the body, so if the articles aren't already being marked as spam, then they won't be marked as spam.

It will break the yenc spec though.

Cheers!

[mbruel]

@loungebob I got some personal issues plus a new work mission but I'll try to find some time to finish this summer. Not much to do to release the v5 ;)

[loungebob]

@loungebob I got some personal issues plus a new work mission but I'll try to find some time to finish this summer. Not much to do to release the v5 ;)

It’s alright, I’ve found ways to work around the issues. Looking forward to the next release. Keep your chin up and remember, it could always be worse.

[loungebob]

@mbruel any updates/news? things might be getting complicated on newsnet.