News aggregator with Python and NewsAPI.

Photo by Roman Kraft on Unsplash

The Problem:

I like to read news about the topics I am exploring. But I will waste time surfing different sites trying to find news articles.

The Solution:

Build a new aggregator or delivery system which provides a maximum number of news articles per topic.

It will contain a description, which serves as an overview of the article, and an URL to the source, which will take me to the original article.

The Implementation:

I will use NewsAPI as the source of the articles and the Telegram bot as a client to interact with this service.

Telegram will accept a keyword or a list of keywords, and it will return a message or messages containing three articles per topic.

The Steps

1. Get a response from the News API
2. Configure and connect the Telegram Bot.
3. Deploy it on Raspberry pi.

1. Get a response from the News API.

Photo by Markus Winkler on Unsplash

News API provides two endpoints to interact.

• /v2/everything: It gathers all information about a specific topic
• /v2/top-headlines: Gets the top headlines based on country and language.

Any successful request will get a JSON file with the news article as a result.

(My) Overview: NewsAPI

I made some substantial changes to the code presented in the overview article about News API. The changes aim to practice the usage of python dataclases and traditional classes.

If you don't want to read the jump, just scroll down I explain what it does in a few lines

The news module contains Dataclass representing the articles and a class representing the call itself

The code is in this Github Gist

Class Aggregator

Represents the request. To create an object for this class I need:

• topics_of_interest topic to search.
• newsapi_key is the News API key.
• from_time and to_time define the timeframe.

get_news()

The function of the Aggregator class. It is used to get news articles.

The response will be a string containing all information, with this structure

{‘topic1’:[{article1},{article2},{article3}],
‘topic2:[{article1},{article2},{article3}] }

To get the articles. I need an object of the class Aggregator and with the object I call the function get_news()

2. Configure the Telegram Bot.

I have some notes about the telegram bot in this article:

(My Notes) Telegram Bot <pyTelegramBotAPI>

Here are the highlights:

1. Get An API key using BotFather.
2. Set some message handlers.
3. The infinity_polling() to keep the bot running.

BotFather

To illustrate the basics, here is an example this is how a simple telegram bot responds “This is an answer to options” when the user type options.

For the News aggregator, I need a Bot that will receive a key-work News Followed by some topics are separated by commas.

News Cardano, Silver, Gas

The format of this input is due to its simplicity of it, I use the key-work News due to the idea to keep updating the bot to make it more user-friendly, like allowing changes in the sources, times, and quantity of articles.

I will provide a link to the Github version, but for now, I will show the code in this way and describe the different parts below.

Basic Bot for the News Aggregator

The code is in this GitHub Gist

Basic set up

I created the Bot object and set the time frame.

#Bot object
bot = telebot.TeleBot(token=os.environ['BOTAPIKEY'])
#fix timeframe
today = datetime.date.today()
older = today - datetime.timedelta(days=4)

Next, I followed the API recommendation and provided support for some basic commands (commands are keywords typed in the chat prefix with`/`). In this case /start, /help

@bot.message_handler(commands=['start', 'help'])
def send_welcome(message):
   """bot start function"""
   bot.reply_to(message, "Type 'News' follow by a key work, you will       get back a list of the latest 3 news")

Now, I have the main body.

@bot.message_handler(func=verify_key)
def bot_get_news(message):
 """Get the news """
# get the topics
 text = message.text.split()
 _, key_words = text[0], text[1:]
# call the object aggregator that contain the topics and the news
 news = Aggregator(
  topics_of_interest=key_words,
  newsapi_key=os.environ['NEWS_API'],
  from_time=older,
  to_time=today
 )
# get the articles
 msg = news.get_news()
 logger.debug(msg=f"getting the news: \n{msg}")
 for topics, news in msg.items():
  bot.send_message(message.chat.id, topics.replace(',', '').upper())
  bot_create_msg(message=message, news=news)

Here there are other functions involved:

1. The Decorator handler call verify_key(), which will verify that the message contains the keyword News follow for at least one topic.
2. The second function is located at the end, bot_create_msg(), which will get the news per topic and separate them into individual messages.

Individual message

The end result will be something like:

example of how the bot works

4. Deploy it on Raspberry pi.

Photo by Stefan Cosma on Unsplash

1. Get the Raspberry Pi and the OS ready.
2. Get git.
3. Clone the Repository from GitHub.
4. Install package.
5. Set environment variables.
6. Run the program.

Get the Raspberry Pi and the OS ready.

I decided to use a Version of Raspbian lite without the desktop, and I will access it by SSH.

Here is a short article made by @Vivienne Encarnacion where she explains how to install the OS and check for free space:

How to do a fresh install of Raspbian OS

Get git

I will use git to get the News aggregator repository.

To get git in the raspberry, I need it to type:

sudo apt-get install git

Checking if git is installed correclty.

Clone the Repository from GitHub.

I keep the code of the New Aggregator with the Telegram Bot in this repository:

GitHub - CubeVic/New_aggregator:

I plan to continue working on the project, especially refactoring the code and providing more options to the bot such as changing the number of news, the sources, and the time frame.

I cloned it using:

git clone https://github.com/CubeVic/New_aggregator.git

Cloning the repository

Install package.

First, I need to install the python package manager.

sudo apt install python3-pip

Second, I installed the package required with

pip install -r requirements.txt

Success installing packages with requirements.txt and pip

Set environment variables.

I am using environment variables for the News API key and for the Bot API key, so I need to configure these keys.

I will create an environment variable for each one like this:

export NEWS_API=<News API>
export BOTAPIKEY=<Bot API key>

There are better ways to do this, but, in my case, this is the simplest one.

Run the program.

Now, I need to run main, and the bot will be running in the Pi.

Bot running on Raspberry Pi

Final Thoughts

1. I am fetching the news for a specific list of sources, those sources, for now, are hardcoded, and an option to modify those sources should be the next step for this bot.
2. The program needs access to the API keys, in this example, I store the keys on the environment variable, but it is better to look for different ways to protect those API keys.
3. I haven't tested the code for vulnerabilities, so a second next step will be to run Bandit against it to get a list of potential vulnerabilities.

Victor Fernandez

• Medium
• My LinkedIn