WARNING: this document is being slowly updated. See the date of each section to get a feel of how up-to-date and reliable is the information in it.
This is the "hacker’s guide" to Newsboat. It describes the overall architecture of the program, the most important design decisions, and some other noteworthy things that don’t belong anywhere else.
This guide assumes that you know about Unix programming, C++, Rust, multithreading, and some more stuff. This is not for end users.
Status of this section: incomplete (January 2021).
This section describes the different classes and their purpose.
class cache: the persistence backend of newsbeuter. It takes rss_item and rss_feed objects and writes them to the cache.db using the SQLite library, respectively reads the content of cache.db and creates rss_item and rss_feed objects.
class colormanager: manages the color configuration. It hooks into the configuration parser, stores the color configuration information and sets them in the view. This is currently a bit ugly, because colormanager knows about the internals of view resp. pb_view. This should be changed in a way that colormanager knows nothing about who gets the configuration, and that the views retrieve this configuration from colormanager. It is derived from config_action_handler, a helper class for the configuration parser.
class configcontainer: manages the normal program configuration. It hooks into the configuration parser and stores the configuration information. Other components can then query the configcontainer for configuration values. It is derived from config_action_handler.
class configparser: parses the configuration file and hands over the results to all hooked config_action_handler objects.
class controller: the controller does a lot of the work within newsbeuter. It is the connector between view and cache. It parses the command line option, controls the configuration parser and hands over data to the view. It also contains code to control the reloading of feeds.
class htmlrenderer: takes a string and renders the HTML it contains into a textual representation. It also extracts URLs from a and img tag for later use.
class keymap: hooks into the configuration parser and manages all the keymapping. Additionally, it generates the keymapping information with hint texts for the view.
class logger: helper class that manages the optional logging to a text file. Logging can be enabled by developers (see below).
class mutex: a C++ wrapper around the pthread mutex.
class reloadthread: a thread that does nothing but starts a reload every n minutes (configurable) for all feeds.
class rss_feed: represents an RSS feed, including RSS url, page link, title, author, description and RSS items (articles). Uses the cache to persist itself. Internally, all text data (especially the title and the author) are stored as UTF-8 strings, but the getters return data that matches the current locale charset setting.
class rss_item: represents an RSS item (article), including link to the article, title, author, publication date and description. Internally, all text data (especially the title, the author and the description) are stored as UTF-8 strings, but the getters return data that matches the current locale charset setting.
class stflpp: a C++ wrapper around STFL. STFL is the ncurses widget library that newsbeuter heavily relies upon.
class thread: a wrapper around Unix pthreads.
class urlreader: manages reading and writing the urls file, including handling of tags.
class utils: contains several static utility functions, such as a tokenizer, the lock file code and a text converter that builds upon iconv.
class view: the class that draws the user interface. It manages a stack of so-called "form actions", each of which represents one dialog. The view class delegates all received user input events to the correct form action.
class formaction: the abstract base class for all form actions. Currently, the following formaction-derived classes exist: - feedlist_formaction - itemlist_formaction - itemview_formaction - urlview_formaction - filebrowser_formaction - help_formaction - selecttag_formaction
class tagsouppullparser: Parses virtually everything that vaguely looks like XML/HTML tags, even when totally invalid, and provides the parsing result as a continuous stream of tokens (tags and text). It is solely used by the htmlrenderer class.
Status of this section: the idea holds (January 2021).
The "classical" text tools, like vim, slrn and mutt, are all configurable solely via text files. Newsbeuter follows the same spirit, especially since the other prominent RSS feed readers for the text console primarily encourage configuration via an often crude user interface within the application itself. The consequence for newsbeuter is: no configuration via the user interface, but solely via configuration files. Text editors are easier to handle than some crude menus that are somehow hard to use.
Status of this section: the idea holds, but Newsboat arguably failed this goal (January 2021).
The problem with user wishes is that too many people demand a possibility to customize this bell or that whistle within newsbeuter. Often, these possibilities only have a very limited purpose, and their value is in no relation to the added complexity of the code. Every customization needs to be tested, and means a lot more testing whenever some related code changes.
The code shouldn’t get too bloated, it should be kept straight-forward and easy to read. With too much customizability, this goal would be in danger.
Status of this section: partially holds, but we’re moving to Rust (January 2021).
C++ has many advantages compared to other programming languages.
-
C++ is backwards-compatible to C. That means we can theoretically use all the C libraries.
-
C++ makes it easier to structure your program in an object-oriented way, and helps maintain inheritance hierarchies without a lot of fuzz.
-
C++ compiles to fast, native code.
-
C++ comes with an extensive standard library (see next section).
-
C++ is widespread.
-
C on Linux/Unix systems does not require any exotic compilers to be installed in order to compile newsbeuter. g (part of GCC) is enough.
These were the reasons why C++ was initially chosen, and it proved to be a useable language during the development process.
Status of this section: mostly holds, but new stuff should be written in Rust if possible (January 2021).
The C++ standard library comes with an extensive set of algorithms and data structures. Developers are encouraged to use especially the data structures, because the available container classes are standardized, their behaviour and usage is well-documented, and makes it possible to keep the overall logic at a pretty high level. More complex things that can only be done in C (like special system calls) /should/ be encapsulated by a wrapper class in order to avoid potential mis-use of low-level functions and data structures. Good examples for wrapping low-level stuff are class rss_feed, class rss_item and class stflpp.
Status of this section: up-to-date (January 2021).
If you want to get a detailed debug log from newsbeuter, you only need to run newsbeuter with special parameters:
newsbeuter -d log.txt -l 6
Some of this output doesn’t make sense very much unless you know the source code, so it’s only helpful for developers.
Status of this section: up-to-date (January 2021).
C++ tests are in the test subdirectory. They’re using
the Catch2 framework. Rust
tests are split: unit tests are in mod tests sub-modules under
rust/libnewsboat/src, while integration tests are under
rust/libnewsboat/tests. Rust’s test runner is multi-threaded, so we use
integration testing when we want to do the checks in a separate process.
In addition to TMPDIR=/dev/shm make -j5 PROFILE=1 check incantation that’s
already explained in the contributing guidelines, you should know about the
NEWSBOAT_RUN_IGNORED_TESTS variable. Like PROFILE, it can be set to 1.
When set, it enables tests that require some additional prerequisites:
-
locales:
-
en_US.UTF-8 -
ru_RU.CP1251 -
ru_RU.KOI8-R