News aggregator with Python and NewsAPI.
The Problem:
I like to read news about the topics I am exploring. But I will waste time surfing different sites trying to find news articles.
The Solution:
Build a new aggregator or delivery system which provides a maximum number of news articles per topic.
It will contain a description, which serves as an overview of the article, and an URL to the source, which will take me to the original article.
The Implementation:
I will use NewsAPI as the source of the articles and the Telegram bot as a client to interact with this service.
Telegram will accept a keyword or a list of keywords, and it will return a message or messages containing three articles per topic.
The Steps
- Get a response from the News API
- Configure and connect the Telegram Bot.
- Deploy it on Raspberry pi.
1. Get a response from the News API.
News API provides two endpoints to interact.
/v2/everything
: It gathers all information about a specific topic/v2/top-headlines
: Gets the top headlines based on country and language.
Any successful request will get a JSON file with the news article as a result.
I made some substantial changes to the code presented in the overview article about News API. The changes aim to practice the usage of python dataclases and traditional classes.
If you don't want to read the jump, just scroll down I explain what it does in a few lines
The code is in this Github Gist
Class Aggregator
Represents the request. To create an object for this class I need:
topics_of_interest
topic to search.newsapi_key
is the News API key.from_time
andto_time
define the timeframe.
get_news()
The function of the Aggregator class. It is used to get news articles.
The response will be a string containing all information, with this structure
{‘topic1’:[{article1},{article2},{article3}],
‘topic2:[{article1},{article2},{article3}] }
To get the articles. I need an object of the class Aggregator and with the object I call the function get_news()
2. Configure the Telegram Bot.
I have some notes about the telegram bot in this article:
Here are the highlights:
- Get An API key using BotFather.
- Set some message handlers.
- The
infinity_polling()
to keep the bot running.
To illustrate the basics, here is an example this is how a simple telegram bot responds “This is an answer to options” when the user type options
.
For the News aggregator, I need a Bot that will receive a key-work News
Followed by some topics are separated by commas.
News Cardano, Silver, Gas
The format of this input is due to its simplicity of it, I use the key-work News due to the idea to keep updating the bot to make it more user-friendly, like allowing changes in the sources, times, and quantity of articles.
I will provide a link to the Github version, but for now, I will show the code in this way and describe the different parts below.
The code is in this GitHub Gist
Basic set up
I created the Bot object and set the time frame.
#Bot object
bot = telebot.TeleBot(token=os.environ['BOTAPIKEY'])#fix timeframe
today = datetime.date.today()
older = today - datetime.timedelta(days=4)
Next, I followed the API recommendation and provided support for some basic commands (commands are keywords typed in the chat prefix with`/`). In this case /start
, /help
@bot.message_handler(commands=['start', 'help'])
def send_welcome(message):
"""bot start function""" bot.reply_to(message, "Type 'News' follow by a key work, you will get back a list of the latest 3 news")
Now, I have the main body.
@bot.message_handler(func=verify_key)
def bot_get_news(message):
"""Get the news """# get the topics
text = message.text.split()
_, key_words = text[0], text[1:]# call the object aggregator that contain the topics and the news
news = Aggregator(
topics_of_interest=key_words,
newsapi_key=os.environ['NEWS_API'],
from_time=older,
to_time=today
)# get the articles
msg = news.get_news()
logger.debug(msg=f"getting the news: \n{msg}")
for topics, news in msg.items():
bot.send_message(message.chat.id, topics.replace(',', '').upper())
bot_create_msg(message=message, news=news)
Here there are other functions involved:
- The Decorator handler call
verify_key()
, which will verify that the message contains the keyword News follow for at least one topic. - The second function is located at the end,
bot_create_msg()
, which will get the news per topic and separate them into individual messages.
The end result will be something like:
4. Deploy it on Raspberry pi.
- Get the Raspberry Pi and the OS ready.
- Get git.
- Clone the Repository from GitHub.
- Install package.
- Set environment variables.
- Run the program.
Get the Raspberry Pi and the OS ready.
I decided to use a Version of Raspbian lite without the desktop, and I will access it by SSH.
Here is a short article made by @Vivienne Encarnacion where she explains how to install the OS and check for free space:
Get git
I will use git to get the News aggregator repository.
To get git in the raspberry, I need it to type:
sudo apt-get install git
Clone the Repository from GitHub.
I keep the code of the New Aggregator with the Telegram Bot in this repository:
I plan to continue working on the project, especially refactoring the code and providing more options to the bot such as changing the number of news, the sources, and the time frame.
I cloned it using:
git clone https://github.com/CubeVic/New_aggregator.git
Install package.
First, I need to install the python package manager.
sudo apt install python3-pip
Second, I installed the package required with
pip install -r requirements.txt
Set environment variables.
I am using environment variables for the News API key and for the Bot API key, so I need to configure these keys.
I will create an environment variable for each one like this:
export NEWS_API=<News API>
export BOTAPIKEY=<Bot API key>
There are better ways to do this, but, in my case, this is the simplest one.
Run the program.
Now, I need to run main, and the bot will be running in the Pi.
Final Thoughts
- I am fetching the news for a specific list of sources, those sources, for now, are hardcoded, and an option to modify those sources should be the next step for this bot.
- The program needs access to the API keys, in this example, I store the keys on the environment variable, but it is better to look for different ways to protect those API keys.
- I haven't tested the code for vulnerabilities, so a second next step will be to run Bandit against it to get a list of potential vulnerabilities.