Tweeper 1.4.1 released, fixed twitter to RSS conversion
Tweeper is a web scraper which converts Twitter and other social media websites to RSS.
On September 7th 2019 version 1.4.1 of Tweeper has been released. The tweeper Debian package has also been updated.
For PHP composer users there is also ao2/tweeper on packagist, which means that tweeper can be installed and run with these commands:
$ composer global require ao2/tweeper $ ~/.config/composer/vendor/bin/tweeper
Version 1.4.1 is a maintenance release with a couple of fixes.
Here are the NEWS file entries:
News for v1.4.1: ================ * Enable cookie handling in cURL to fix scraping twitter.com * Update User-Agent version to fix scraping hashtag pages on twitter.com
![[Valid RSS]](https://ao2.it/sites/default/files/valid-rss-rogers.png "Validate my RSS feed")
Comments
Did twitter break tweeper
Did twitter break tweeper recently?
php ../git/tweeper/tweeper.php https://twitter.com/business | tee business PHP Warning: Error 801: Tag svg invalid in Entity, line 138 in /home/aas/git/tweeper/src/Tweeper.php on line 257 PHP Warning: Error 801: Tag g invalid in Entity, line 138 in /home/aas/git/tweeper/src/Tweeper.php on line 257 PHP Warning: Error 801: Tag path invalid in Entity, line 138 in /home/aas/git/tweeper/src/Tweeper.php on line 257 <?xml version="1.0"?> <rss version="2.0" xml:base="https://twitter.com"> <channel> <generator>Tweeper</generator> <title>Twitter / </title> <link>https://twitter.com/business</link> <description/> </channel> </rss>Yes, it looks like
Yes, it looks like twitter.com now serves content via json to modern browsers and parses and renders it on the client side.
Adjusting the
User-Agentheader to mimic an older browser seems to fix this and does not break the other sites supported by Tweeper, see https://git.ao2.it/tweeper.git/commitdiff/da4568f5a2d24e0933d44b16b5ef180095c42dab.Post new comment